Storage issues OpenStack London
2016-09-18 01:50 After some further close monitoring of the involved systems we deem the service stable and the issues completely resolved. Should you have any issues as a result of tonights events that you cannot solve yourselves, or you have any questions, don’t hesitate to contact our support department on email@example.com.
2016-09-18 01:30 Due to a lack of progress there has been a lack of updates here. We have finally managed to resolve the issues and storage is back online. We have also verified this from inside VMs that were running before the issues, and are still running. No VMs have been rebooted. The nature of the issues will probably have looked like a total lock of all I/O bound tasks inside the VMs, where all reads and writes would just have blocked for the 3,5 hours this outage lasted.
Most services are resilient enough to cope with even such extreme conditions. For example webservers is most likely going to be just fine. However there might be cases where services inside VMs have crashed, and as we do not generally have access inside customers VMs we cannot identify and resolve such issues.
We now consider this issue resolved, but will monitor the involved systems very closely for a while in order to have a firm verification of that.
2016-09-17 23:50 It is taking much longer than expected to resolve the issues.
2016-09-17 22:30 We are still working on resolving the issues.
2016-09-17 22:05 We have confirmed the issues are related to the storage systems, and are now working on resolving the issues.
2016-09-17 22:00 We are experiencing storage issues in OpenStack London, technicians are investigating.