Select a region
The reasons behind the week-long IT issues that have plagued all States staff and services this week have still not been determined.
As engineers employed directly by Agilisys, and those contracted to help them over the past week, continue to try and work out exactly why the States back up system didn't kick in, the number of questions about what caused the problem are growing by the day.
Everything IT related within the States - from schools' wifi systems and email accounts to benefit payments, online portals and more - went offline last Friday morning with an air conditioning unit being blamed.
The first sign of a problem came early on 25 November when a warning system alerted engineers that a server room at Sir Charles Frossard House was getting 'too hot'.
Pictured: An air conditioning unit was blamed for the initial fault at Sir Charles Frossard House last week.
The warning system is set to go off at 25c. By the time engineers got to the server room just 20 minutes later, the temperature had already reached 44.1c.
A third-party contractor was called in to try and help cool the equipment and rectify the proble. By the time they arrived the server room was now up to 48c.
It has now been confirmed that this server room, with the faulty air conditioner unit is just one of three critical data storage systems owned by the States.
They are: at Frossard House, at Edward T Wheadon House, and a third site at an unknown location.
The States communications team has today confirmed that the main Equipment Room is the one at Frossard House, which suffered the initial fault. It holds around 500 servers.
The back-up Equipment Room is the one at Wheadon House. Both of these server rooms hold live data which is replicated across both sites.
States engineers do an "incremental back up of all data every evening and weekly full back up to a third site"
The issue last Friday morning caused such serious problems because the main server room at Frossard House went into ‘preservation mode’ as it reheated. That meant it automatically shut down to "protect itself and the live data in the system".
Engineers attempted a ‘fail over’ which would have meant the running of all services being switched over to the back-up equipment room at Wheadon House. However that failed.
Pictured: The States second or 'back up' server room is at Edward T Wheadon House in Le Truchot.
One week on, the States communications team said: "It is unclear at this stage why the main equipment room did not successfully switch to the back-up and this requires further investigation, which are underway."
After a week of questions being asked and mainly left unanswered, the States has also assured us that "further information will be provided to the public once those investigations have been completed."
It won't just be the public who are waiting for answers, as the island's most senior politician and civil servant are both also waiting for any further information.
Deputy Peter Ferbrache, President of the Policy & Resources Committee, said the situation this week has been unacceptable.
"Clearly the circumstances of the last week are not good enough, despite the significant amount of work from engineers who have been trying to get systems back up and staff across services managing any disruption. Being a week in and not fully operational is concerning and we need answers as quickly as possible.
"Our Committee, States Members and the wider community understandably want answers in language we can understand, and importantly reassurance about what will happen to make sure we never see a repeat. We will be expecting those answers after things are back to normal."
Mark de Garis, States of Guernsey Head of the Public Service, also said the situation was "unacceptable" and he apologised for all of the problems caused by the outages.
"An outage such as we have experienced over the last week is unacceptable and urgent actions are being taken to fully understand how this happened," he said.
Pictured(l-r): Deputy Peter Ferbrache and Mark de Garis.
"The States of Guernsey runs a large number of systems supporting a huge range of services; for example we have bespoke systems in areas such as children’s services, the ports,Beau Sejour, social security systems responsible for the payment of benefits and many online services through gov.gg.
"This makes the recovery of systems extremely complex. Agilisys engineers and third-party contractors have worked tirelessly over the last eight days to restore services as quickly as humanly possible and I would like to publicly thank them.
"This however remains a completely unacceptable situation and I want to apologise to all our service users who have been affected by the disruption. We will provide further information as soon as the detailed investigations are complete."
In confirming some of the facts around the issues of the last week, the States also again assured the public that all data held by the island's various government organisations was protected at all times:
States IT outage to last all week
Deputies ban ormering at night but IT failures delay key runway debate
Comments
Comments on this story express the views of the commentator only, not Bailiwick Publishing. We are unable to guarantee the accuracy of any of those comments.