Update - The storage cluster continues to operate with stable performance for the majority of workloads. Regarding the subset of volumes requiring further intervention, we are closely coordinating with our storage vendor to execute comprehensive integrity scans and fully map the scope of residual inconsistencies.
In parallel, the vendor is currently finalizing the development of automated remediation workflows, modeled on successful manual recovery procedures validated on a limited data subset earlier today. We are closely monitoring this development and will only authorize the execution of these workflows once the solution has been rigorously validated. We will provide further details as we prepare to transition from these ongoing scans to the active recovery phase.
Dec 18, 2025 - 23:00 UTC
In parallel, the vendor is currently finalizing the development of automated remediation workflows, modeled on successful manual recovery procedures validated on a limited data subset earlier today. We are closely monitoring this development and will only authorize the execution of these workflows once the solution has been rigorously validated. We will provide further details as we prepare to transition from these ongoing scans to the active recovery phase.
Dec 18, 2025 - 23:00 UTC
Update - The majority of customers should now observe a return to normal operations and consistent performance within the eu-iceland1-a region. While general cluster stability has been restored, engineering teams are continuing focused recovery efforts for a limited subset of customers who remain more deeply impacted by residual metadata inconsistencies.
We are maintaining close oversight of service health and will continue to work directly with our storage vendor to finalize remediation for the remaining impacted volumes. Further updates will be provided as the recovery process nears full completion
Dec 18, 2025 - 18:37 UTC
We are maintaining close oversight of service health and will continue to work directly with our storage vendor to finalize remediation for the remaining impacted volumes. Further updates will be provided as the recovery process nears full completion
Dec 18, 2025 - 18:37 UTC
Identified - Engineering teams have implemented configuration changes to isolate background recovery processes from active customer workloads. Following these adjustments, we are observing signs of stabilization in cluster performance and a reduction in the intermittent connectivity fluctuations reported earlier.
We are continuing to monitor service health closely. Please note that while stability has improved, further maintenance operations may be required to finalize the recovery efforts. We are currently evaluating the necessity and timing of these potential steps with the vendor and will provide updates as plans are finalized.
Dec 17, 2025 - 23:11 UTC
We are continuing to monitor service health closely. Please note that while stability has improved, further maintenance operations may be required to finalize the recovery efforts. We are currently evaluating the necessity and timing of these potential steps with the vendor and will provide updates as plans are finalized.
Dec 17, 2025 - 23:11 UTC
Investigating - We are continuing to address the ongoing service instability affecting shared volumes in the eu-iceland1-a region. While engineering teams continue the previously reported background recovery efforts, we are observing intermittent performance fluctuations and connectivity drops, which may result in transient errors for active workloads.
We are actively engaged with our storage vendor to diagnose the specific drivers of this instability and are working to restore consistent performance to the cluster.
Dec 17, 2025 - 21:22 UTC
We are actively engaged with our storage vendor to diagnose the specific drivers of this instability and are working to restore consistent performance to the cluster.
Dec 17, 2025 - 21:22 UTC
Identified - Hello Team, during our monitoring we are observing VM operation failures, as well as shared volume create/update/delete failures due to ongoing impact to the storage control plane. Some customers may still observe issues when accessing existing volumes. A small subset of specific files or directories may remain temporarily inaccessible or return Input/Output (I/O) errors.
We are actively engaged with our vendor and pursuing further investigation on this matter.
Dec 17, 2025 - 02:53 UTC
We are actively engaged with our vendor and pursuing further investigation on this matter.
Dec 17, 2025 - 02:53 UTC
Monitoring - Engineering teams have completed the critical phase of the integrity verification process and have lifted the broad protective maintenance restrictions. As a result, service availability has been restored for the majority of volumes and data paths in the eu-iceland1-a region.
However, a small subset of specific files or directories may remain temporarily inaccessible or return Input/Output (I/O) errors. We have transitioned to a monitoring phase and teams are actively working to restore full access to these remaining items. We will provide a final confirmation once the incident is fully resolved.
Dec 17, 2025 - 02:05 UTC
However, a small subset of specific files or directories may remain temporarily inaccessible or return Input/Output (I/O) errors. We have transitioned to a monitoring phase and teams are actively working to restore full access to these remaining items. We will provide a final confirmation once the incident is fully resolved.
Dec 17, 2025 - 02:05 UTC
Identified - Engineering teams are currently executing a targeted integrity verification process to permanently resolve the metadata inconsistencies reported earlier. To ensure strict data safety during this operation, the storage cluster has entered a protective maintenance state that temporarily restricts access to prevent modification while checks are running.
As a result of these active safety measures, customers in eu-iceland1-a may experience service unavailability, including mount failures or Input/Output (I/O) errors, for specific volumes, directories, or files. Please be assured that these restrictions are intentional precautions to preserve data integrity. We expect to lift these protective measures once the verification process is complete.
Dec 17, 2025 - 00:49 UTC
As a result of these active safety measures, customers in eu-iceland1-a may experience service unavailability, including mount failures or Input/Output (I/O) errors, for specific volumes, directories, or files. Please be assured that these restrictions are intentional precautions to preserve data integrity. We expect to lift these protective measures once the verification process is complete.
Dec 17, 2025 - 00:49 UTC
Update - Engineering teams are continuing to actively remediate specific storage metadata inconsistencies and stabilize network connectivity within the eu-iceland1-a region.
During this window, customers may continue to experience intermittent write errors, latency spikes, or brief connectivity drops. We are prioritizing work to restore full service performance and expect to provide further details within approximately 2 hours.
Dec 16, 2025 - 22:43 UTC
During this window, customers may continue to experience intermittent write errors, latency spikes, or brief connectivity drops. We are prioritizing work to restore full service performance and expect to provide further details within approximately 2 hours.
Dec 16, 2025 - 22:43 UTC
Update - We are continuing to observe impact related to the storage performance degradation in eu-iceland1-a. Our engineering teams remain actively engaged with our vendor in focused remediation efforts and are working to address residual intermittent slowness for some customers utilizing Shared Storage services.
We will provide another update as soon as we have further developments. Thank you for your continued patience
Dec 16, 2025 - 18:53 UTC
We will provide another update as soon as we have further developments. Thank you for your continued patience
Dec 16, 2025 - 18:53 UTC
Update - We have implemented a fix for the storage performance issues in eu-iceland1-a. Storage latency and throughput have returned to normal operating levels. We are actively monitoring the service to ensure continued stability.
Dec 16, 2025 - 18:26 UTC
Dec 16, 2025 - 18:26 UTC
Update - We are expanding the scope of impact for this incident. Customers will observe significant slowness and intermittent access issues when utilizing Shared Storage services in eu-iceland1-a region.
We will provide another update as soon as we have further developments. Thank you for your patience as we work to restore full service as quickly as possible.
Dec 16, 2025 - 17:36 UTC
We will provide another update as soon as we have further developments. Thank you for your patience as we work to restore full service as quickly as possible.
Dec 16, 2025 - 17:36 UTC
Investigating - We are currently investigating slowness when using Shared Storage in eu-iceland1-a.
We are working with our storage vendor to resolve the issue as soon as possible.
Dec 16, 2025 - 17:13 UTC
We are working with our storage vendor to resolve the issue as soon as possible.
Dec 16, 2025 - 17:13 UTC
GPU Virtual Machines
Operational
us-east1
Operational
us-northcentral1
Operational
us-southcentral1
Operational
eu-iceland1
Operational
Networking
Operational
Wide Area Networking (WAN)
Operational
VPC Networking
Operational
Infiniband Networks
Operational
Datacenter Networking
Operational
Storage
Degraded Performance
Shared Disks
Degraded Performance
Persistent Storage
Operational
Container Registry
Operational
Orchestration
Operational
Crusoe Managed Kubernetes (CMK)
Operational
API
Operational
UI
Operational
Operational
Degraded Performance
Partial Outage
Major Outage
Maintenance
Major outage
Partial outage
No downtime recorded on this day.
No data exists for this day.
had a major outage.
had a partial outage.
Scheduled Maintenance
Crusoe Cloud Database Migration Maintenance (All Regions) Jan 8, 2026 02:00-02:30 UTC
Scheduled -
Crusoe Cloud will be undergoing planned Database Upgrade maintenance in all regions. During this maintenance window, the only impact you will experience is a brief 30 minute interruption to all control plane operations. This includes actions such as VM Stop, Start, Delete, or other similar operation using the UI or CLI. All other network services and running workloads will remain operational throughout the maintenance window. If you notice any other issues, please reach out to support@crusoe.ai
Nov 24, 2025 - 23:04 UTC
Nov 24, 2025 - 23:04 UTC