Openstack storage issue in Stockholm. Updated
2017-05-18 12:50: We have been monitoring storage performance since last week and haven’t seen any further issues arise or disturbances occur. We therefore set this issue as cleared.
2017-05-11 10:20: For customers with VMs that still has issues with corrupt file systems, we recommend rebooting or running fsck to restore the file system or snapshot restore/disaster recovery in that order if necessary. If this still wont fix your issue please create a ticket by sending an email to firstname.lastname@example.org with your account info and what VM is effected.
2017-05-11 08:12: We have monitored the services overnight and everything seems stable at the moment. We continue to monitor and will make a new status evaluation at noon.
2017-05-10 13:48: We believe we have found the cause of the disruption and technicians are working to take proactive measures to ensure that they do not repeat themselves.
2017-05-10 13:48: Underlying storage seems stable at the moment and issues have been resolved. Technicians are currently working to restore affected customer VMs.
2017-05-10 13:06: Issues still persists. Technicians are awaiting spare parts to reinforce hardware capacity.
2017-05-10 12:22: Certain problems have been resolved but issues still exist and technical staff is working on resolving them.
2017-05-10 11:10: We believe we have found the cause of the issues and technical staff is working on resolving them.
2017-05-10 10:10: We are still experiencing problems with storage performance in Stockholm. The issues are being investigated.
2017-05-09 18:50 Most of the issues have been resolved, but some VMs are still having issues with corrupt file systems. We recommend rebooting, running fsck to restore the file system or snapshot restore/disaster recovery in that order if necessary.
2017-05-09 14:20 Higher load is causing some VMs to have reduced performance when writing to storage. Investigation ongoing.
2017-05-09 10:30 We have noticed a slight decrease of performance on the storage system, we are currently working on improving the performance. However, the system is stable overall.
2017-05-09 00:30 The storage system is fully back online, the actual downtime was two instances of about 10 minutes each. We will, however, continue to closely watch the behavior of the system.
2017-05-08 23:50 We are going to have to perform some emergency maintenance on the storage system for OpenStack in Stockholm. Expected downtime is at most 30 minutes.
2017-05-08 14:25: The virtual machines are online again with normal operations. We will continue to monitor the situation.
2017-05-08 14:10: We have identified an issue with one of storage nodes in Stockholm. Virtual machines running on this storage are having issues at the moment. Troubleshooting has begun.