Wednesday 07 June 2023
Select a region

Questions growing as IT issues continue

Questions growing as IT issues continue

Friday 02 December 2022

Questions growing as IT issues continue

Friday 02 December 2022

The reasons behind the week-long IT issues that have plagued all States staff and services this week have still not been determined.

As engineers employed directly by Agilisys, and those contracted to help them over the past week, continue to try and work out exactly why the States back up system didn't kick in, the number of questions about what caused the problem are growing by the day.

Everything IT related within the States - from schools' wifi systems and email accounts to benefit payments, online portals and more - went offline last Friday morning with an air conditioning unit being blamed.

The first sign of a problem came early on 25 November when a warning system alerted engineers that a server room at Sir Charles Frossard House was getting 'too hot'.

Frossard house air conditioner

Pictured: An air conditioning unit was blamed for the initial fault at Sir Charles Frossard House last week.

The warning system is set to go off at 25c. By the time engineers got to the server room just 20 minutes later, the temperature had already reached 44.1c.

A third-party contractor was called in to try and help cool the equipment and rectify the proble. By the time they arrived the server room was now up to 48c.

It has now been confirmed that this server room, with the faulty air conditioner unit is just one of three critical data storage systems owned by the States.

They are: at Frossard House, at Edward T Wheadon House, and a third site at an unknown location.

The States communications team has today confirmed that the main Equipment Room is the one at Frossard House, which suffered the initial fault. It holds around 500 servers.

The back-up Equipment Room is the one at Wheadon House. Both of these server rooms hold live data which is replicated across both sites.

States engineers do an "incremental back up of all data every evening and weekly full back up to a third site"

The issue last Friday morning caused such serious problems because the main server room at Frossard House went into ‘preservation mode’ as it reheated. That meant it automatically shut down to "protect itself and the live data in the system".

Engineers attempted a ‘fail over’ which would have meant the running of all services being switched over to the back-up equipment room at Wheadon House. However that failed.

Wheadon House

Pictured: The States second or 'back up' server room is at Edward T Wheadon House in Le Truchot.

One week on, the States communications team said: "It is unclear at this stage why the main equipment room did not successfully switch to the back-up and this requires further investigation, which are underway."

After a week of questions being asked and mainly left unanswered, the States has also assured us that "further information will be provided to the public once those investigations have been completed."

It won't just be the public who are waiting for answers, as the island's most senior politician and civil servant are both also waiting for any further information.

Deputy Peter Ferbrache, President of the Policy & Resources Committee, said the situation this week has been unacceptable.

"Clearly the circumstances of the last week are not good enough, despite the significant amount of work from engineers who have been trying to get systems back up and staff across services managing any disruption. Being a week in and not fully operational is concerning and we need answers as quickly as possible.

"Our Committee, States Members and the wider community understandably want answers in language we can understand, and importantly reassurance about what will happen to make sure we never see a repeat. We will be expecting those answers after things are back to normal."

Mark de Garis, States of Guernsey Head of the Public Service, also said the situation was "unacceptable" and he apologised for all of the problems caused by the outages.

"An outage such as we have experienced over the last week is unacceptable and urgent actions are being taken to fully understand how this happened," he said.

Peter Ferbrache mark de garis

Pictured(l-r): Deputy Peter Ferbrache and Mark de Garis.

"The States of Guernsey runs a large number of systems supporting a huge range of services; for example we have bespoke systems in areas such as children’s services, the ports,Beau Sejour, social security systems responsible for the payment of benefits and many online services through

"This makes the recovery of systems extremely complex. Agilisys engineers and third-party contractors have worked tirelessly over the last eight days to restore services as quickly as humanly possible and I would like to publicly thank them.

"This however remains a completely unacceptable situation and I want to apologise to all our service users who have been affected by the disruption. We will provide further information as soon as the detailed investigations are complete."

In confirming some of the facts around the issues of the last week, the States also again assured the public that all data held by the island's various government organisations was protected at all times: 

  • The States of Guernsey has a main Equipment Room (which has circa 500 servers running) housed at Sir Charles Frossard House.
  • A back-up Equipment Room is housed at Edward T Wheadon House.
  • The live data is replicated across both sites.
  • The States also does an incremental back up of all data every evening and weekly full back up to a third site.
  • A warning system is in place to alert the States if the equipment room gets too hot. It is set to 25 degrees and alerted us there was an issue on Friday morning (25 November 2022).
  • IT engineers arrived at the site within 20 minutes, confirmed the air conditioning systems had failed and the room temperature was at 44.1 degrees so immediately carried out health checks on all equipment.
  • Third-party contractor were notified immediately and dispatched cooling engineers to site.
  • The room temperature had increased to 48 degrees upon the arrival of the third- party contractor.
  • The main system went into ‘preservation mode’, which is a safety mechanism for it to automatically shut down and protect itself and the live data in the system.
  • This successfully safeguarded all live data held by the States of Guernsey, however it did cause significant performance issues across the States IT network.
  • However, despite a ‘fail over’ being attempted, switchover of the services to the back-up equipment room at Edward T Wheadon House failed.
  • It is unclear at this stage why the main equipment room did not successfully switch to the back-up and this requires further investigation, which are underway.
  • Further information will be provided to the public once those investigations have been completed.


States IT outage to last all week

IT issues update

Deputies ban ormering at night but IT failures delay key runway debate

Sign up to newsletter



Once your comment has been submitted, it won’t appear immediately. There is no need to submit it more than once. Comments are published at the discretion of Bailiwick Publishing, and will include your username.

There are no comments for this article.

To place a comment please login

You have landed on the Bailiwick Express website, however it appears you are based in . Would you like to stay on the site, or visit the site?