The circuit was leased from an unnamed wholesale network supplier. When the circuit went down, it triggered a traffic storm that overloaded the carrier’s network.
The issue started on Monday when T-Mobile customers reported having difficulty with both phone calls and text messages, specifically on the carrier’s (Voice over LTE) network.
Data connections were stable as was calling and texting on Sprint’s network. Customers on other carriers reported issues, but most of those complaints were from customers trying to reach someone on the T-Mobile network.
As soon as the outage was recognized, T-Mobile assigned hundreds of engineers and worked for hours with partner companies and vendors.
The problem was finally traced back to a failure in a leased fiber circuit.
Situated at a provider in the Southeast, the circuit failure caused an IP traffic storm that spread from this Southeast area and overloaded the carrier’s VoLTE network.
T-Mobile said these types of failures are common, and the company has redundancy built into its network that’ll reroute traffic. In this incident, the redundancy also failed.
T-Mobile, in a statement, assured customers that it is adding safeguards to prevent this outage from happening again.
It also is reviewing what happened during the blackout to identify the cause of the overload and how to manage such a large spike in traffic without causing a widespread outage.