This report highlights the operational challenges of scaling AI-integrated services and global infrastructure. It provides insights into managing model-backed dependencies, handling cross-cloud network issues, and mitigating traffic spikes to maintain high availability for developer tools.
In December, we experienced five incidents that resulted in degraded performance across GitHub services.
December 08 19:51 UTC (lasting 1 hour and 15 minutes)
Between November 26, 2025, at 02:24 UTC and December 8, 2025 at 20:26 UTC, enterprise administrators experienced a disruption when viewing agent session activities in the Enterprise AI Controls page. During this period, users were unable to list agent session activity in the AI Controls view. This did not impact viewing agent session activity in audit logs, directly navigating to individual agent session logs, or otherwise managing AI agents.
The issue was caused by a misconfiguration in a change deployed on November 25 that unintentionally prevented data from being published to an internal Kafka topic responsible for feeding the AI Controls page with agent session activity information.
The problem was identified and mitigated on December 8 by correcting the configuration issue. GitHub is improving monitoring for data pipeline dependencies and enhancing pre-deployment validation to catch configuration issues before they reach production.
December 15 17:43 UTC (lasting 39 minutes)
On December 15, 2025, between 15:15 UTC and 18:22 UTC, Copilot Code Review experienced a service degradation that caused 46.97% of pull request review requests to fail, requiring users to re-request a review. Impacted users saw the error message: “Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.” The remaining requests completed successfully.
The degradation was caused by elevated response times in an internal, model-backed dependency, which led to request timeouts and backpressure in the review processing pipeline, resulting in sustained queue growth and failed review completion.
We mitigated the issue by temporarily bypassing fix suggestions to reduce latency, increasing worker capacity to drain the backlog, and rolling out a model configuration change that reduced end-to-end latency. Queue depth and request success rates returned to normal and remained stable through peak traffic.
Following the incident, we increased baseline worker capacity, added instrumentation for worker utilization and queue health, and are improving automatic load-shedding, fallback behavior, and alerting to reduce time to detection and mitigation for similar issues.
December 18 16:33 UTC (lasting 1 hour and 8 minutes)
On December 18, 2025, from 08:15 UTC to 17:11 UTC, some GitHub Actions runners experienced intermittent timeouts for Github API calls, which led to failures during runner setup and workflow execution. This was caused by network packet loss between runners in the West US region and one of GitHub’s edge sites. Approximately 1.5% of jobs on larger and standard hosted runners in the West US region (0.28% of all Actions jobs) were impacted during this period.
By 17:11 UTC, all traffic was routed away from the affected edge site, mitigating the timeouts. We are working to improve early detection of cross-cloud connectivity issues and faster mitigation paths to reduce the impact of similar issues in the future.
December 18 17:36 UTC (lasting 1 hour and 33 minutes)
On December 18, 2025, between 16:25 UTC and 19:09 UTC, the service underlying Copilot policies was degraded and users, organizations, and enterprises were not able to update any policies related to Copilot. No other GitHub services, including other Copilot services, were impacted. This was due to a database migration causing a schema drift.
We mitigated the incident by synchronizing the schema. We have hardened the service to make sure schema drift does not cause any further incidents, and we will investigate improvements in our deployment pipeline to shorten time to mitigation in the future.
December 22 22:31 UTC (lasting 1 hour and 46 minutes)
On December 22, 2025, between 22:01 UTC and 22:32 UTC, unauthenticated requests to github.com were degraded, resulting in slow or timed out page loads and API requests. Unauthenticated requests from Actions jobs, such as release downloads, were also impacted. Authenticated traffic was not impacted. This was due to a severe spike in traffic, primarily to search endpoints.
Our immediate response focused on identifying and mitigating the source of the traffic increase, which along with automated traffic management restored full service for our users.
We improved limiters for load to relevant endpoints and are continuing work to more proactively identify these large changes in traffic volume, improve resilience in critical request flows, and improve our time to mitigation.
Follow our status page for real-time updates on status changes and post-incident recaps. To learn more about what we’re working on, check out the engineering section on the GitHub Blog.
The post GitHub Availability Report: December 2025 appeared first on The GitHub Blog.
Continue reading on the original blog to support the author
Read full articleThis report highlights how complex dependencies—like telemetry, caching, and security policies—can trigger cascading failures. It provides valuable lessons on the importance of robust monitoring, automated rollbacks, and the need for resilient proxy layers in large-scale distributed systems.
This post highlights how rapid scaling and architectural coupling can turn localized issues into platform-wide outages. It provides lessons on managing cache TTLs, the risks of latent configuration errors in failover systems, and the necessity of robust load-shedding mechanisms.
This shift transforms AI from a chat interface into programmable infrastructure. By embedding execution engines into apps, developers can build resilient, context-aware systems that handle complex multi-step tasks without brittle, hard-coded logic or custom orchestration layers.
This architectural shift eliminates common failure modes in high-availability setups where search indexes could become locked or corrupted during upgrades. By using native Cross Cluster Replication, engineers gain a more resilient, easier-to-maintain search infrastructure.
