I wanted to write to you to apologise for the disruption some of our customers have experienced over the last few weeks.
Unfortunately, on Tuesday 13th February we had a failure in part of our core network. There should have been an automatic switch from one data centre to the other, resulting in no loss of service for customers, however this did not happen and as a result some of our customers lost service.
Our engineers mobilised quickly, and we were able to get most customers back online through the course of Tuesday morning, however, there were additional changes required to ensure our service was robust and this created further disruption for some of our customers over the course of Wednesday and Thursday.
We also had a completely unrelated incident on Tuesday 20th February, which affected the network and resulted in a loss of service. Our engineers were able to identify the problem quickly, which was caused by a network partner, and resolve it within an hour and restore service.
As a result of these incidents, we are conducting a thorough review of the network, to ensure this does not happen again and improve our resilience. This will inevitably mean that we will need to make changes on the network. These will be made overnight, and we will provide advance warning to any customers that may be affected to keep disruption to a minimum.
In addition, we are looking at how we can improve our customer communications during such incidents. One immediate action is to add a network status page to our website. This will provide you with details of any incidents we have on the network and their status alongside information about any planned maintenance work. This will go live within the next week.
We are committed to providing a fast and reliable service and investing to ensure we can deliver this for all our customers. The last few weeks have been out of the ordinary, however it was not good enough, and you have my commitment as the founder of toob that we will be learning from the incident and making the necessary investment to improve.
My sincere apologies for the inconvenience and thank you for your patience.
Nick
CEO & Founder