Background and Impact

Buried under the news from Google I/O this week is a significant mishap involving Google Cloud, the tech giant’s competitor to Amazon Web Services (AWS). The incident involved the accidental deletion of a large customer account, UniSuper, an Australian pension fund managing $135 billion in assets with 647,000 members. This account deletion included all backups stored on Google Cloud. Fortunately, UniSuper had backups with a different provider and managed to recover its data. However, the downtime began on May 2, and full service restoration wasn't achieved until May 15.

 

Incident Overview

UniSuper's website now provides detailed accounts of this administrative nightmare. On May 8, a page titled "A Joint Statement from UniSuper CEO Peter Chun and Google Cloud CEO Thomas Kurian" was posted. The statement revealed that the disruption was due to an unprecedented sequence of events. An inadvertent misconfiguration during the provisioning of UniSuper’s Private Cloud services led to the deletion of its Private Cloud subscription. Thomas Kurian, Google Cloud CEO, described this as a unique, isolated incident that had never occurred with any of Google Cloud's clients globally. Kurian emphasized that this should not have happened and assured that measures have been taken to prevent such occurrences in the future.

 

Detailed Timeline of Events

May 2: The incident begins, leading to significant downtime for UniSuper.

May 8: A joint statement from UniSuper and Google Cloud is released, explaining the cause of the disruption.

May 9: Signs of recovery emerge as logins for online UniSuper accounts start functioning. However, account balances may not reflect transactions processed during the outage.

May 13: The mobile app starts working again, though balances are still not up-to-date. UniSuper assures that they are processing transactions as quickly as possible.

May 15: UniSuper confirms that all member-facing services have been fully restored, including their retirement calculators.

 

Joint Statement Analysis

The joint statement, though intended to clarify the incident, remains vague and uses terminology that doesn't align with standard Google Cloud products. This lack of precision suggests that the statement might have been authored entirely by UniSuper, with Google simply signing off on it. This has raised concerns about transparency and the technical specifics of the error.

 

Response and Measures

Following the incident, both UniSuper and Google Cloud have assured their users that steps are being taken to ensure such an error does not recur. Google Cloud has identified the events leading to this disruption and has implemented measures to prevent a repeat. Despite these assurances, the incident has undeniably shaken confidence in Google Cloud's reliability, particularly among current and potential customers who are closely observing how Google handles the fallout.

 

Recovery and User Impact

UniSuper's phased recovery efforts included various updates on their website detailing the restoration of services. Starting with basic login capabilities, UniSuper progressively reinstated features of their online and mobile services. The final update on May 15 marked the full restoration of all services, including the critical retirement calculators used by their members.

Industry Repercussions

This incident highlights the potential risks associated with cloud services and underscores the importance of having robust backup solutions with multiple providers. It also serves as a cautionary tale for other companies relying heavily on cloud infrastructure, emphasizing the need for thorough due diligence and contingency planning.

 

Conclusion

Google Cloud’s accidental deletion of UniSuper's account is one of the most significant blunders in the company’s history. While the immediate crisis has been managed and services restored, the incident leaves lingering questions about cloud security and the robustness of preventive measures. As the cloud computing industry continues to grow, such incidents remind stakeholders of the critical need for transparency, reliable service, and effective incident management protocols.