This report highlights the complexities of scaling a massive platform while migrating to cloud infrastructure. It provides insights into managing database hotspots, handling cascading failures in service meshes, and the critical trade-offs between availability and feature velocity.
While we continue to make progress, August proved to be a challenging month for availability. You can read more about these incidents in a blog post we published last month. We are aggressively investing in both architectural improvements and moving to Azure, which will give us more capacity. Meanwhile, we continue to see significant growth on our platform. We are prioritizing the most impactful work while minimizing risk, but as these incidents in August show, we cannot completely eliminate risk.
Ultimately, all the work needs to be done, and incidents give us an opportunity to adjust our priorities. As repair items from these incidents, we’ve made significant improvements to our capacity monitoring and management, retry policies that led to bigger impact, and resiliency improvements to core services. We’ve also continued to make great progress across many durable work streams.
On August 11, GitHub ran a production MySQL primary from Azure for the first time. Client-observed write impact was minimal, and no customer impact occurred in the transition. We repeated the pattern with two more primaries on August 27. We have further primaries scheduled over the coming weeks, increasing in complexity as we learn from each failover.
Read traffic also reached new highs. Reads from migrated services peaked at 60.4%, while reads from GitHub’s monolith peaked at 64.3% in Azure. Git reads reached 54%.
Away from the regional migration, the 24-table authentication-core cohort moved off GitHub’s oldest shared database, mysql1, removing approximately one million queries per second from its replicas. Separate query-hygiene changes removed another 120,000 queries per second and eliminated approximately 59,000 seconds of wasted database work per hour.
GitHub Actions gained additional capacity while longer-term isolation work continues. Job-routing changes moved 33% of jobs from a constrained production cluster to spare capacity, reducing peak cache CPU utilization from 98% to 80% and adding an estimated three months of headroom. This is a near-term containment measure, not the finish line; the August outage reinforced the need for more durable capacity and isolation, which continues.
Pull request isolation work continued. In addition to unauthenticated traffic already being served, now authenticated reads for the first production cohort reached 100%.
Investments in Git overload protection served 6.4% more traffic while improving 95th-percentile duration by 24% and maximum delay by 78%. Broader load-shedding protections at the edge also made progress, enabling levers that can protect GitHub under unexpected load—in fact, these protections were used in mitigating the aforementioned incidents in August.
We also improved monitoring and telemetry. Pull request monitoring now measures merge, review, and comment failures independently, where high read volume could otherwise mask a failing write path. On August 21, automated high impact incident detection began combining customer-support signals with service telemetry. API monitoring was also recalibrated and validated over 30 days, reducing noise and improving signal quality. These changes improve detection and response.
The next month of work includes moving the next database primaries, continuing to migrate services and corresponding traffic to Azure, chipping away at database health particularly on shared-databases, adding more automation around capacity management and auto-scaling, and extending dependency-failure handling across more of the pull request experience.
This principle continues to guide us: availability, then capacity, then features.
The incident began with a routine deployment to an internal GitHub Actions service that processes incoming events and turns them into actions jobs. The deployment’s contents were not at fault (we rolled it back to confirm this); instead, replacing pods during the rollout briefly reduced capacity in one site and pushed the remaining sites past their limit as traffic shifted to them. Impact was heaviest through the middle hours of the incident, when a large share of actions workflow runs were failing to start or complete.
The affected actions services were running close to their capacity and concurrency limits. A routine deployment that briefly reduced the number of running pods was enough to exhaust available headroom. This caused service mesh sidecars to experience CPU throttling and out-of-memory restarts, which then cascaded into cache, DNS, and API errors across multiple clusters. The ingress service mesh for these services had limited headroom so it could not absorb the temporary loss of capacity during the deployment.
As the core services recovered, a latent bug in the job-assignment path made recovery slower: runners were handed jobs that had already been revoked, then got stuck retrying them instead of picking up valid work, which created a self-amplifying backlog.
A new peak in traffic pushed one datacenter’s load balancers past their limits. A service-mesh sidecar reached its concurrency limit and did not scale up.
As requests backed up, several of the datacenter’s load-balancer nodes exhausted their network flow limits, which degraded the shared gateway authentication path and produced widespread authentication latency and failures across the many services that route through that datacenter.
A latent client retry bug sharply amplified traffic to one internal authentication endpoint, which slowed recovery for the Copilot Token Service. The core weakness was that our service-mesh sidecar did not scale up, together with our retry behavior, and clients were not bound enough to keep a partial degradation from amplifying into a broader overload.
During the incident window, the Copilot cloud agent task was impacted. The tasks themselves still ran to completion, so no work was lost. Once processing caught up the correct status and results appeared. Waiting a short time, or checking back a little later, would have shown the up-to-date state.
Across the incident, at least 54 organizations experienced Copilot Cloud Agent task status and results lagging well above their own normal level. Per-minute customer-facing impact peaked at 37.5% of measured task-status activity.
Copilot cloud agent stores the status and results of each agent task in a managed cloud database. One region of that database suffered a provider-side outage, and the calls that read and write task status in the affected region began failing and running slow.
The processors that stream task-status updates into that database then fell behind as database latency climbed. Their throughput is bounded by a fixed number of processing partitions sized for normal latency plus some headroom. The latency in this incident went well past that headroom, causing the backlog of task-status updates to grow instead of clearing. A storage configuration on the database also made the affected region slow to fail over, so the first failover attempts did not take effect, and recovery took longer than expected.
Dependent services like Copilot code review and some GitHub Pages deployments that run on top of actions, were impacted during the incident window.
In most cases, delayed runs started once the backlog drained, and runs that failed to start succeeded when re-run after the incident. A small set of runs that were created during the earliest part of the incident could not be recovered by re-running and had to be started fresh.
The quick summary is that our shared infrastructure services have not kept up with our month-over-month actions growth and peak load.
A burst of incoming events arrived on top of an already-high load and pushed the database past its tipping point. Query times climbed, and the database primary saturated.
With the database overloaded, the internal service that turns incoming events into runner assignments could not keep up, so actions runs failed to start and began queuing well past their normal start time.
Failing over the database primary helped only partially. The throttles used to relieve inbound load were initially set slightly too high to fully protect the database, so recovery had to be ramped up slowly and manually.
There was no automatic circuit breaker to throttle inbound actions load when the database showed early signs of stress, so the protective throttling had to be applied and tuned manually during the incident. This is one of the learnings from this incident.
Customers who had configuration to use the Kimi K3 model were impacted by this incident. Customers who were using other models or switched to using other models were not impacted.
Copilot offers a choice of AI models. One of them, Kimi K3, is served by an upstream model provider.
That provider had a serving degradation that caused a large share of Kimi K3 requests to fail with errors. Because the problem was with the upstream provider, requests that used other models—and requests made with the Auto setting, which routed to a different model—were not affected.
A steady share of Kimi K3 requests kept failing until the provider’s mitigation took hold. At peak, more than half of the requests using Kimi K3 were failing.
Follow our status page for real-time updates on status changes and post-incident recaps. To learn more about what we’re working on, check out the engineering section on the GitHub Blog.
The post GitHub availability report: August 2026 appeared first on The GitHub Blog.
Continue reading on the original blog to support the author
Read full articleThis report illustrates the complex reality of migrating a massive monolith to the cloud. It highlights how architectural isolation, dedicated services, and automated safeguards are essential for maintaining availability while scaling and moving away from legacy shared infrastructure.
This report provides a transparent look at large-scale infrastructure migration and service extraction. It highlights the trade-offs between speed and stability, the importance of edge rate-limiting, and how to handle database contention in a massive monolith-to-microservices transition.
This report highlights the challenges of scaling a massive monolith under AI-driven traffic growth. It provides a blueprint for reliability through infrastructure migration, service decomposition, and the implementation of automated circuit breakers to prevent cascading failures.
This architectural shift eliminates common failure modes in high-availability setups where search indexes could become locked or corrupted during upgrades. By using native Cross Cluster Replication, engineers gain a more resilient, easier-to-maintain search infrastructure.
