We have rolled out a fix to all regions and customers can now connect to their VMs. Please reach out to support@crusoecloud.com if you have any questions or concerns.
Posted Jun 11, 2024 - 01:55 UTC
Update
We have begun rolling out a fix to production VMs, and select customers can begin accessing their workloads once again. We expect all customers to have connectivity restored by 02:00 UTC.
Posted Jun 11, 2024 - 01:35 UTC
Monitoring
We have validated a fix in our development environment and are working to deploy the fix to production services. We are estimating that this will be deployed by 01:30 UTC .
Posted Jun 11, 2024 - 00:43 UTC
Update
We identified the root cause of the issue and were able to reproduce in our development environment. We are continuing to work on a fix in in this environment before rolling out changes to production services.
We will provide another update at 00:30 UTC.
Posted Jun 10, 2024 - 23:47 UTC
Update
We are testing a fix to address this issue in a development environment. We will proceed with the rollback to production databases once testing is complete.
We will provide another update by 23:45 UTC.
Posted Jun 10, 2024 - 23:16 UTC
Update
We are still investigating the root cause of the issue and are in the process of rolling back our control plane database. It appears that VMs in the us-east1-a region are more likely to be affected by this issue.
Posted Jun 10, 2024 - 22:33 UTC
Identified
We are still investigating the source of the lost IP associations, but are working to rollback our control plane database to a previous snapshot. During this time, all control plane functions will not be operational.
We will be providing another update by 22:30 UTC.
Posted Jun 10, 2024 - 21:59 UTC
Investigating
We are currently investigating an issue where VMs have lost access to their Public IPs. We will provide an update again once we have more details.
Posted Jun 10, 2024 - 21:26 UTC
This incident affected: Networking (Wide Area Networking (WAN)).