Emergency update due to possible vulnerability in KVM/Xen
We are closing this ticket as we have completed all the work required after the final reboots. We thank you again for your patience as we secured the complete environment as well each individual server.
Sunday and the morning of Monday we have continued to go through host node after host node and also verified each VM in our environments. As we have thousands of VMs it takes some time with the verification process. Some VMs have also required manual work to make sure they function properly.
We expect to finish the full process by Tuesday morning and do various cleanup work that does not affect any customers during Tuesday.
Work continues as planned. A majority of servers have been handled but we expect to continue to finalize over the next 24 to 48 hours. While most of our customers will not be affected anymore – some will as we reboot some host nodes as well as VMs. Thank you again for your patience.
Updates has been going well and the foundation of City Cloud and affected virtual environments have been patched and properly updated to allow to remove the potential threats. Over the next 24 hours there will still be a large portion of VMs that will potentially be rebooted to gain access to the newly updated hypervisors and related OS. We have looked for ways to improve the process to minimize down time.
As your machines can potentially be rebooted – please make sure all your processes start up automatically or else you will have to logon to start up your applications after reboot.
It has been noted that there are serious security holes in KVM/Xen hypervisors which can potentially compromise virtual machines created with those hypervisors. As keeping your data safe is paramount to us we are performing emergency updates in our complete environment that runs KVM (City Cloud). This is a problem affecting most cloud providers as well as those running Linux and KVM/Xen in general.
This weekend we will patch all our virtual host nodes as well as possibly reboot virtual machines as it is required in some cases. Please expect some disturbances this weekend in your environments as we go through these procedures to secure your virtual machines from possible serious threat.
We will update here when we near completion.
Thank you for your patience!