Resolved -
This incident has been resolved. All affected compute hosts and customer virtual machines in us-east1-a have been restored.
May 20, 21:09 UTC
Update -
Remediation is ongoing and the number of impacted customers continues to narrow. Engineering remains actively engaged, and we will provide a further update as the work advances.
May 20, 18:55 UTC
Update -
Engineering continues to make progress on remediation since the last status post. Work remains ongoing for the customers still in progress, We will provide a further update as recovery advances.
May 20, 17:55 UTC
Update -
Engineering has restored the majority of affected services in us-east1-a and is in the final stages of remediation. A subset of customers remain impacted and are being prioritised for recovery. We will provide a further update shortly.
May 20, 16:30 UTC
Update -
Work to restore service in us-east1-a continues. Engineering has made significant progress, with the large majority of affected compute hosts now recovered. A small number of hosts remain under active remediation. We will provide a further update shortly.
May 20, 15:35 UTC
Update -
Work to restore service in us-east1-a continues, with engineering making steady progress across affected compute hosts. A subset of hosts remain impacted as remediation work advances. We will provide a further update shortly.
May 20, 14:27 UTC
Update -
Engineering continues to make progress on the issue in us-east1-a, though recovery is proceeding more slowly than initially anticipated. A subset of compute hosts in the region remains impacted while remediation work continues. We will provide a further update as recovery advances.
May 20, 13:27 UTC
Update -
Work to restore service is ongoing as our teams continue to drive remediation in us-east1-a. Engineering remains engaged across affected compute hosts and we will provide a further update shortly.
May 20, 12:37 UTC
Update -
Engineering continues to work on the issue in us-east1-a. A subset of compute hosts in the region remains impacted while remediation steps are being applied. We will provide a further update once additional progress can be reported
May 20, 11:28 UTC
Update -
Remediation in us-east1-a is ongoing and continuing to progress. Additional virtual machines have returned to service; however, a subset of compute hosts remains impacted and the region is not yet fully restored. Engineering remains engaged and further updates will follow.
May 20, 10:32 UTC
Update -
We continue to see progress in us-east1-a as engineering works through mitigation. Additional virtual machines have returned to service, although a subset of compute hosts remains impacted, and the issue is not yet fully mitigated. Engineering remains actively engaged, and we will provide further updates as recovery continues.
May 20, 09:28 UTC
Update -
We are seeing partial recovery in us-east1-a following mitigation steps applied by engineering. A subset of affected virtual machines have returned to service.
Engineering is continuing to validate the mitigation and work toward full restoration. A subset of compute hosts remains impacted at this time. We will provide further updates as progress continues.
May 20, 08:25 UTC
Identified -
We are continuing to work on the issue affecting us-east1-a. Engineering has identified a likely cause and mitigation is in progress. A subset of compute hosts remains unavailable, and Managed Kubernetes and Container Registry operations in the region continue to be impacted.
We will continue to provide updates as remediation progresses.
May 20, 06:19 UTC
Update -
We are continuing to investigate this issue.
May 20, 03:51 UTC
Investigating -
We are investigating a networking issue in the us-east1-a region affecting connectivity to a subset of compute hosts. Affected virtual machines may appear in an unspecified state or be unreachable over the network. Managed Kubernetes clusters in the region may also report nodes as Not Ready. Container Registry operations in the region, including image push/pull and repository management, are also impacted.
Engineering is engaged and actively working to identify the cause and restore service. We will provide updates as we have more information.
May 20, 01:37 UTC