Limited Outage – City Cloud
We currently have a limited outage in City Cloud. Our technicians are looking in to the matter.
All Vm´s are not affected, the issues are limited to a few nodes.
We are still working hard to solve the issues in City Cloud.
The problems are now located and we are in the process of restarting the VM´s. All Vm´s will not restart at once, it can take some time for all of the affected ones to get back online.
Many of the machines are now fully functional again, we are doimg all we can to get the rest back online.
It might take longer than we expected to get all the VM´s back online due to queue. We will update again shortly.
Some background information about the outage. As many of you might know, we have had trouble with one of our storage nodes – Gluster (from Red Hat). This node is once again causing problems. This system has known issues and a month ago, we started moving VM´s from it, 700 at the moment. The node is scheduled to be taken offline durin 2012. Today we decided to move even more machines since they, like it or not, are affected and offline at the moment. It´s better to get them up and running on a functioning node. It takes quite some time to move all the data but we will try to move the remaining VM´s as soon as we can.
Even if you log in and don´t see your Vm in the control panel, it´s there. They cant be shown to you due to the function of Gluster. There is no risk of losing data or VM´s for that matter. We will get the affected VM´s up and running and are doing so every minute that passes. It will however take some time due to the amount of data that we are working with. More information will follow, thanks for your patience.
There are still queues to get the affected Vm´s up and running, out technicians are working hard to get them up manually. Right now, many of the affected VMs are back online and more and more are on the way.
We are now working on two things. To get the Gluster node up and running to buy us some more time and also moving VM´s from this node. There are still 150 Vm´s left to get back online and depending on their disk sizes it can take several more hours to get them back online. More information will follow.
No changes. We are still trying to get the Gluster node back online. We thank you for your patience.
System testing has shown some progress. Our technicans are now automating the revival of VM´s to get them up as quickly as possible.
All affected Vm´s should now be up and running except for a few (10) that will be moved to a new storage node during the night. If you still have any issues with your VM, please contact our support department for help.
The last VM´s are now moved and back online. All Vm´s that were running before the outage should be back to their previous state. If you have any problems with your VM, please contact firstname.lastname@example.org/com for further investigation.
We´d like to add that the VM´s located on our Gluster node will be moved during the next week. About 200 VM´s (out of the 1000 that we had there in the beginning of the year) . More info will follow during next week. We thank you for your patience totay.