City Network Status

OnlineBackup Service is unavailable

2018-11-12 13:30 We are pleased to inform our OnlineBackup customers that we now have working procedures in place for restoring backups from before this outage. It is however not going to be possible to do that within the OnlineBackup clients installed on your desktops and servers. If you need a backup from before 2018-09-24, please contact us at support@citycloud.com and we will help you retrieve the backup data.

As it since 2018-11-05 is possible to take new backups and restore those, as well as now also possible to retrieve old backup data we will now close this outage. Should you still have any kind of issues related to OnlineBackup, don’t hesitate to contact us at support@citycloud.com.

 

2018-11-09 11:00 Work has been ongoing this week into setting up the required framework to facilitate restores of backups from before the outage. This is a complex process and have proven to be very time-consuming. We will thus not be able to complete this phase during this week. Work will continue over the weekend and based on the current status we should be able to complete this during the first half of the next week.

2018-11-05 15:05 All customers should now be able to perform new backups with OnlineBackup. If your backups aren’t being taken, start by checking that the OnlineBackup client is running on your desktop or server. There is also a connectivity check in the web interface for the client with which you can check that you can reach the backup servers. If your backups is still not working, please navigate to the backup progress for a failing backup job in the client web interface, and check to see if there’s an error message mentioned there, and then contact support@citycloud.com, include eventual error message in the email.

We will now put more resources into making it possible to restore old backups from before the outage. As it’s early in the process we can’t give any estimates, but our goal is that we should be able to restore old backups before end of this week.

2018-11-05 12:45 Most customers should now be able to perform new backups with OnlineBackup. A small batch of customers remain where they might not be able to perform a backup.

2018-11-05 09:30 The fix for the server-side issues have been implemented during the weekend, but due to the way the service works server-side, there are server-side configuration for each backup client which needs to be updated. This is in progress since about 15 minutes ago, and once completed for each client, that client will be able to take new backups.

2018-11-02 16:45 Due to issues with the server software for the OnlineBackup service we will not be able to achieve our goal of enabling new backups today. We have identified the issues, and developed a workaround which have been successfully tested. We are now working on implementing this workaround and once that’s done, we should be able to bring the service online again.

2018-11-02 We have unfortunately experienced non-resolvable issues with the storage system used for OnlineBackup. This has resulted in the need to setup a completely new storage system. The data on the old storage system, albeit completely intact is going to take very long time to transfer to the new storage system. Despite setting up high capacity links directly between the storage systems it is just not feasible to transfer all the data.

We have thus decided to start up the service without the data, in order to enable our customers to take new backups. In the meantime we are working with our vendors on developing measures to retrieve specific backup sets from the old storage system on request from customers. We want to stress that the data is still intact and still readable.

We expect to have the service up and able to receive new backups before the end of today. Do however note that since the new storage system doesn’t contain any data, all backups will be full backup sets again, regardless of backup schema settings.

 

2018-10-30 6AM UTC (07:00 CET) – The work to bring online backup available again is still ongoing.. We will update this page on the progress.

2018-10-26 1PM UTC (14:00 CET) – We are now installing new storage nodes to our online backup service and will moving all data from the old backup storage to the new. We will update this page on the progress.

2018-10-24 2PM UTC (15:00 CET) – Hardware replacement is ongoing, and we work hard to get the services up and running as soon as possible.

2018-10-24 9PM UTC (10:00 CET) – Onlinebackup service is down for the moment. Troubleshooting is ongoing.

2018-09-24 – OnlineBackup currently suffers from decreased hardware performance resulting in failed schedules. A plan is carried out to resolve these issues and improve on overall performance. This will take time due to the shear amount of data and number of clients. We ask for your understanding in this matter until fully resolved. Any questions in this matter is best sent to support@citycloud.com for further handling.