What were the exact details of the outage?
The outage was caused by one of our network suppliers, Tele2. It commenced around 02:00 (GMT) on 31/01/2019 and had fully restored by 16:30 (GMT) on the same day. It affected a small proportion of CSL services with Tele2 SIMs. Dual path devices with a secondary IP or PSTN path were still able to send alarms.
What was the cause of the incident and how was it resolved?
The cause was planned maintenance activity that Tele2 undertook early that morning, which disrupted their core communications network across Europe. That planned maintenance was to upgrade Tele2’s core IP communications network. During this work a routing error was introduced, initially blocking some of their network traffic. This caused a cascade effect that eventually swamped their core IP network and resulted in the loss of services that CSL experienced. Once the issue had been identified and remedied, the services were restored in ordered blocks of connections with CSL prioritised as part of that activity. Throughout the incident, CSL were constantly engaged at director level with Tele2 as they went through the process to restore service.
How was the outage communicated?
Outage notifications were sent to ARCs and information was shared with Installers via e-letters, our social media platforms and the CSL website. Our Technical Support and Sales Teams also had this information to advise Installers.
How does this incident differ from a similar incident with Tele2 in August 2018?
CSL are requesting an in-depth explanation from Tele2 as to how last week’s incident could have happened following the corrective work that took place in August. Initial conclusions have revealed that this issue was caused by Tele2’s internal core network which didn’t benefit from the network enhancements that took place within its IoT Network. However, lessons from that incident were used in the controlled resolution process which did result in much quicker restoration of service. The Tele2 executive team are meeting with CSL Directors at our head office today as we continue to review the incident.
Our apologies and what steps have been put in place to reduce future risk?
We understand the impact on your business and your customers. We will assess and continue to work with Tele2 to fully understand what exactly went wrong and what action will be undertaken to avoid a re-occurrence. We apologise again for any inconvenience caused and thank you for your patience whilst we investigated.