'Malfunctioning clock' sparked JT outage

Thursday 30 July 2020

A “malfunctioning clock” was the root cause of a six-hour JT network blackout that caused difficulties with 999 calls, a new report by the telecoms firm has said.

The telecoms provider’s network went down for a total of six hours on Sunday 12 July, during which time ATMS crashed, customers were unable to make calls and suffered difficulties with broadband and internet services.

999 calls were also unable to be made via landlines, but remained possible in some cases. 

Having investigated the cause of the major incident, JT released its findings in a report yesterday – just as Jersey’s competition watchdog launched an independent probe into the technical issue, as well as governance and security at the telecoms company.

JT said that the failure was triggered by the malfunctioning of a clock connected to the network’s 100 IP routers.

Pictured: 999 calls were not possible from landlines during the outage. 

At 18:55 on 12 July, one generated a wrong date (27/11/2000), which sparked what JT’s Chief Technology and Information Officer Thierry Berthouloux described as a “sequence of events that was almost impossible to foresee."

Mr Berthouloux said JT investigated whether this could have been the result of malicious activity, but had ruled out this possibility.

Instead, the likely cause was a hardware failure that returned the clock to factory settings. 

With mobile and internet communications hit by the outage, he said JT lost access to its corporate arm, making it difficult to keep customers informed.

Mr Berthouloux said that the need for “more robust customer communications during service incidents” was one of “many learnings” following the outage.

Speaking yesterday, JT CEO Graeme Millar described the failure as “the worst in JT’s 132-year history”, saying that the network had already “taken the steps needed to prevent any repetition”.

He added that the firm’s CFO will be conducting a review of the effectiveness of these steps.

Pictured: JT CEO Graeme Millar said the outage was the worst JT had experienced in over 100 years. 

However, due to the impact on 999 services, the competition authority has nonetheless decided to launch its own, independent review into what happened and has appointed telecoms experts Cognitio Consultants to conduct it.

At the time of the outage, the Jersey Competition Regulatory Authority (JCRA) had already been completing a separate investigation into system failures with the 999 service provided by JT and Sure earlier this year.

However, the six-hour outage in early July led the authority to decide that a more in-depth probe would be needed.

Cognitio’s investigation is expected to last between two and four weeks.

The cost of the independent investigation, which will be covered by JT, is understood to be in the region of £120,000.