A major carrier experienced a catastrophic problem recently, a problem that others can learn from. The carrier in question is Telstra, from Australia, and it affected a huge number of its users.
A missed software update brought down a major carrier’s network
So, what happened? A rather small maintenance omission caused havoc. Telstra said that the lack of a software update on a key time-mapping system is to blame all of this. Its maintenance teams were also unaware of a design change that affected how it would reset.
A mobile outage last Friday affected 45% of all calls and data sessions on Wednesday. The network went down shortly before 4:30 AM local time. Telstra did say that it did not lack redundancy in its network, but that such redundancy did not prevent this outage.
The glitch reset the time to 2006
One of Telstra’s three network time protocol (NTP) servers, which are designed to ensure the systems have the correct time, had reset to 2006. The server was identified as a Microchip model called SSU 2000. It was manufactured in 2011 and costs $30,000 to replace.
Telstra has three of these in Sydney, Melbourne, and Perth. It was still under support from Scientific Devices, and it was confirmed that it was not a hardware problem. What was the problem then? A software update that was not applied by Telstra.
The problem started during maintenance to replace faulty backup power. Telstra shut down and restarted the server in Melbourne, and due to an “underlying software configuration” within the server, the thing restarted with the wrong 2006 date.
The incorrect date slowly spread through the network
That incorrect date rippled slowly but surely across the network, which caused authentication certificates on other servers to become invalid. All that caused havoc across Telstra’s network.
The carrier apologized for the mistake by stating that they’re aware that they let their customers down and fell short of what people rightly expected from them.
The company promised that it will investigate why the design change was not documented, and why the software update was not completed.
The post How One Unapplied Update Brought Down a Major Carrier’s Network for Millions of Users appeared first on Android Headlines.
