This report offers critical insights into distributed systems resilience, dependency management, and incident response. Engineers can learn from these real-world outages to build more robust, fault-tolerant services, emphasizing proactive measures and graceful degradation strategies.
In October, we experienced four incidents that resulted in degraded performance across GitHub services.
October 09 14:45 UTC (lasting 1 hour and 55 minutes)
On October 9, 2025, between 14:35 UTC and 15:21 UTC, a network device in maintenance mode that was undergoing repairs was brought back into production before repairs were fully completed. Network traffic traversing this device experienced significant packet loss.
Authenticated users of the github.com UI experienced increased latency during the first 5 minutes of the incident. API users experienced up to 7.3% error rates, after which it stabilized to about 0.05% until mitigated. Actions service experienced 24% of runs being delayed for an average of 13 minutes. Large File Storage (LFS) requests experienced minimally increased error rate, with 0.038% of requests erroring.
To prevent similar issues, we are enhancing the validation process for device repairs of this category.
October 17 13:11 UTC (lasting 1 hour and 1 minute)
On October 17, 2025, between 12:51 UTC and 14:01 UTC, mobile push notifications failed to be delivered for a total duration of 70 minutes. This affected github.com and GitHub Enterprise Cloud in all regions. The disruption was related to an erroneous configuration change to cloud resources used for mobile push notification delivery.
We are reviewing our procedures and management of these cloud resources to prevent such an incident in the future.
October 20 08:56 UTC (lasting 2 hours and 5 minutes)
On October 20, 2025, between 08:05 UTC and 10:50 UTC the Codespaces service was degraded, with users experiencing failures creating new codespaces and resuming existing ones. On average, the error rate for codespace creation was 39.5% and peaked at 71% of requests to the service during the incident window. Resume operations averaged 23.4% error rate with a peak of 46%. This was due to a cascading failure triggered by an outage in a third-party dependency required to build devcontainer images.
The impact was mitigated when the third-party dependency recovered.
We are evaluating options to ensure this dependency is no longer a critical path in our container build process and improving our monitoring and alerting to reduce the detection time of similar issues in the future.
October 29 16:17 UTC (lasting 6 hours and 58 minutes)
On October 29, 2025, between 14:07 UTC and 23:15 UTC, GitHub experienced service degradation due to a widespread outage at a third-party provider. Codespaces users faced severe connection issues, with error rates averaging 90% and peaking at 100% across all regions during the incident. GitHub Actions larger hosted runners were also impacted, with 0.5% of workflows and nearly 10% of large runner jobs failing or delayed beyond five minutes. Actions impact recovered by 20:40 UTC. The GitHub Enterprise Importer service was impacted as well, causing migration failures during git push operations and significant delays in processing migrations. Additionally, new trials for GitHub Enterprise Cloud with Data Residency were delayed, and Copilot Metrics API downloads were unavailable, resulting in approximately 100 failed requests until recovery began around 20:25 UTC.
Mitigations were applied throughout the incident to reduce impact, but full recovery was only achieved once the service provider resolved its outage. GitHub is now focused on reducing critical path dependencies on external providers and implementing strategies to gracefully degrade services during similar events, aiming to improve resilience against future outages.
Follow our status page for real-time updates on status changes and post-incident recaps. To learn more about what we’re working on, check out the engineering section on the GitHub Blog.
The post GitHub Availability Report: October 2025 appeared first on The GitHub Blog.
Continue reading on the original blog to support the author
Read full articleThis report highlights how complex dependencies—like telemetry, caching, and security policies—can trigger cascading failures. It provides valuable lessons on the importance of robust monitoring, automated rollbacks, and the need for resilient proxy layers in large-scale distributed systems.
This post highlights how rapid scaling and architectural coupling can turn localized issues into platform-wide outages. It provides lessons on managing cache TTLs, the risks of latent configuration errors in failover systems, and the necessity of robust load-shedding mechanisms.
This architectural shift eliminates common failure modes in high-availability setups where search indexes could become locked or corrupted during upgrades. By using native Cross Cluster Replication, engineers gain a more resilient, easier-to-maintain search infrastructure.
This report highlights the operational challenges of scaling AI-integrated services and global infrastructure. It provides insights into managing model-backed dependencies, handling cross-cloud network issues, and mitigating traffic spikes to maintain high availability for developer tools.
