This report illustrates the complex reality of migrating a massive monolith to the cloud. It highlights how architectural isolation, dedicated services, and automated safeguards are essential for maintaining availability while scaling and moving away from legacy shared infrastructure.
The GitHub Actions incident on Thursday, August 6, was unacceptable in both its impact and particularity of its duration. Availability continues to be our top priority across all of GitHub. However, with this incident, we have fallen short of our commitments to you. We know how heavily customers rely on actions, and a prolonged outage like this one has a real impact on your productivity and on your trust in us.
We continue to work through a deeper root cause analysis (RCA) on the incident, as there were many aspects in play that we want to fully understand before calling the investigation complete. We’ll update the public summary when our investigation is complete, and we will include the complete details in our August availability post to be published in September.
Aside from immediate repair items discovered through our investigation, we are accelerating our architectural roadmap in GitHub Actions, aligned to our ongoing efforts around isolation, resiliency, and scale.
It’s worth noting that the GitHub Actions service at the core of the aforementioned incident is still fully running in our data centers, a contributing factor to the lack of capacity we experienced. While the majority of actions runs on Azure, we hadn’t yet prioritized migrating launch service, the component that bridges the monolith to actions, due to its generally asynchronous nature and ability to queue work in response to issues. Unfortunately, as outlined in the public summary, cascading failures led to an unacceptable delay in recovery. This is why we are accelerating our move of GitHub Actions to Azure, where we will have more headroom and capabilities to absorb spikes.
On our broader efforts, last month, we shared how a deliberate pause and stronger stability controls changed the way we move production traffic into Azure. In July, those controls allowed us to resume that work with greater confidence while continuing to reduce shared dependencies across GitHub.
The short version of July: GitHub is becoming less dependent on shared infrastructure and individual datacenter locations, giving us more capacity to absorb growth and making failures easier to isolate. This month, more than half of monolith read traffic ran in Azure Central US, authentication data began leaving our oldest shared database, and dedicated services removed substantial load from that shared path.
GitHub can now serve a larger share of customer requests from independent Azure capacity, reducing reliance on any single datacenter while preserving performance. Monolith read traffic served from Azure Central US peaked at 52.75% on July 28—the first time we consistently remained above the halfway line. Git traffic in Azure reached 47%, up from 43% in June, and 29% of all repositories now have a second replica in Central US, making failover less disruptive when a region degrades. Just as important, the stability validation process introduced after May’s incident is now part of every major traffic expansion, helping us increase capacity without increasing customer risk.
We also reduced shared failure points behind critical customer workflows. The first authentication tables moved from our oldest shared database to dedicated infrastructure, proving the migration pattern for the remaining work. Authentication and permission checks now place substantially less pressure on that shared path: at peak, the dedicated user service offloads more than one million queries per second, while 80% of a major authorization lookup has moved to the isolated service path. Repository content traffic now runs fully from Central US on dedicated infrastructure, and the dedicated pull request service, which already serves anonymous traffic, reached 99.87% parity with the monolith for authenticated reads as we progressively roll all traffic to it. Together, these changes make it less likely that pressure or failures in one part of GitHub will affect unrelated customer activity.
GitHub can now absorb more workload growth before shared infrastructure becomes a source of degradation that affects customers. One production change cut total query time on the artifacts table in half, while caching in the Git authorization path reduced authorization service load by 18.2%, even as request volume grew. Search is also less exposed to the capacity constraints that contributed to earlier incidents: all production search workloads now serve from Central US with additional headroom during demand or infrastructure stress.
We are also changing how we measure and operate reliability. In addition to infrastructure health, we are increasingly measuring the health of important customer workflows such as pull requests so teams can identify degradation earlier. We are continuing to replace high-risk manual production activity with automation, review controls, and operational safeguards, so customer experience is less dependent on perfect human execution.
Crossing 50% is the midpoint, not the finish. The next phase is about building enough independent Azure capacity to serve all production traffic and, ultimately, withstand the loss of a region without failure. This quarter, we are targeting 70% of read traffic and 30% of write traffic in Central US while bringing every production service online there. Moving database primaries will unlock write traffic; a second Azure region will provide the foundation for regional resilience. We now have a line of sight to get dotcom production traffic out of our datacenters by the end of CY2026.
The eight incident write-ups that follow are the other half of this picture—what the system did well, what it did not, and what we have already changed as a result. The principle continues to guide us: availability, then capacity, then features.
July 8, 2026 (lasting 7 hours and 4 minutes)
On July 8, 2026, between 15:07 and 22:13 UTC, multiple GitHub services—including the Web UI, REST API, GraphQL API, Actions, Packages, Copilot, and Git operations—were unavailable across data-resident Enterprise Cloud environments and returned 5xx errors. Affected users experienced page-load and login failures, failed API requests, queued or rejected Actions workflow runs, and unavailable package registry endpoints. During the peak hour, approximately 84% of active tenants across the affected production environments had a majority of their requests fail, and the peak 5xx error rate reached approximately 96% in the most affected environment.
Our automated monitoring detected elevated 5xx errors within approximately 19 minutes of the first impact.
An automated infrastructure metadata process changed a runtime configuration value across virtual machines in these affected environments. A safeguard that normally prevents this value from changing on running machines did not apply to this update. The incorrect value disrupted service discovery and left traffic routers without available backends. We mitigated the incident by restoring the correct configuration values in each of the affected environments and allowing services to re-register. Recovery took several hours because the values had to be corrected and validated across many machines. A separate infrastructure component exhausted its available memory during the recovery and required additional mitigation, extending the recovery duration. We are enforcing metadata immutability, removing unsafe overwrite behavior, and improving staged rollouts. In addition, we are investing in improving monitoring for empty traffic-router backend pools; and developing safer fleet-wide recovery tooling. These changes are intended to prevent recurrence and reduce detection and mitigation for similar issues in the future.
July 9, 2026 (lasting 9 hours and 18 minutes)
On July 9, 2026, between 03:29 and 13:39 UTC, GitHub Actions experienced delayed and failed job starts on GitHub-hosted runners. Some Pages builds, Copilot Cloud Agent jobs and Copilot Code Review jobs were also delayed or failed because they depend on GitHub Actions for job execution. The incident was caused by an unhealthy state in a backend data service responsible for provisioning hosted runners, preventing runner acquisition for a subset of workloads. The shard serving the highest-volume runner workload became overloaded and could not synchronize reliably across regions, preventing runner acquisition for affected workloads. During most of the incident, approximately 8% of workflow runs on hosted runners were delayed by more than 5 minutes, while roughly 2% failed to start.
We restored the health of the backend data replication system, allowing provisioning to recover and the accumulated workflow backlog to drain. Service performance then returned to expected levels. The recovery surge briefly placed additional pressure on Actions and dependent services, but the backlog cleared and service performance returned to expected levels by 13:39 UTC.
GitHub has completed several improvements to monitoring, operational guidance, and workload management following this incident. Additional work is underway to better distribute demand, protect services during recovery, and reduce delays when traffic increases suddenly. Longer-term changes will further improve data resilience and service capacity. These efforts are intended to reduce the likelihood and impact of similar incidents.
July 16, 2026 (lasting 3 hours and 7 minutes)
On July 16, 2026, between 08:50 and 09:50 UTC, the GitHub MCP Server’s web_search tool experienced elevated failures. The average error rate was 42% and peaked at 82% of requests to the tool. Other GitHub MCP Server tools were unaffected. This was caused by degradation at an upstream web search provider.
The incident was mitigated when the upstream provider recovered, after which we confirmed that the tool’s success rate had returned to normal.
GitHub has improved request handling, monitoring, and response guidance for the web search tool: every search now runs under an overall time budget, so a failing provider cannot leave a request running indefinitely, monitoring now tracks the tool’s reliability specifically, and new internal runbooks help responders diagnose provider-side failures and judge when a single degraded tool warrants a public incident.
Additional safeguards are underway, including a circuit breaker that will report search as unavailable once failures persist, rather than letting every request fail on its own. These changes make a provider outage faster to diagnose and clearer to customers, but they cannot return search results while a provider is unavailable.
That requires a second, independent source of results, since the tool had no alternative to fall back on. We are therefore evaluating backup options to help maintain service during future provider interruptions.
July 19, 2026 (lasting 2 hour and 11 minutes)
On July 19, 2026, between 18:00 and 20:11 UTC, a broad DNS reconfiguration caused significant service degradation across github.com and all data residency environments. Customers experienced elevated failed requests across services including GitHub Actions, webhooks, and Copilot. Impact was greater in our data residency environments, where the affected internal service discovery mechanism is more heavily relied upon. The peak regional 5xx error rate was 9.9%, and webhook deliveries in one affected environment were interrupted for approximately 40 minutes.
At 17:50 UTC, a database connectivity issue affected our internal DNS control plane. The resulting incomplete data was incorrectly interpreted by automated reconfiguration of DNS servers across the fleet and disrupted internal service discovery. As cached DNS records expired, some services could no longer resolve internal addresses, causing failed requests.
Automated monitoring detected customer impact within approximately one minute. Recovery began at approximately 18:54 UTC as database connectivity returned. Customer impact ended at 19:26 UTC as DNS forwarding and caches repopulated, and the incident fully resolved at 20:11 UTC.
Following the incident, we deployed safeguards that preserve the last known good DNS configuration, reject incomplete data, and prevent large destructive changes. We also reduced the duration of cached negative DNS responses and improved monitoring for DNS control-plane database availability.
In addition, we are continuing to audit all automated DNS reconfiguration paths and adding read-only database fallback when the primary is unavailable to ensure incomplete data cannot cause destructive changes. These improvements are intended to prevent recurrence and reduce detection and mitigation time for similar issues.
July 19, 2026 (lasting 5 hours and 10 minutes)
Between July 19, 2026 at 23:05 UTC and July 20, 2026 at 03:55 UTC, GitHub Actions experienced delayed and failed job starts for self-hosted and larger runners. Jobs using standard GitHub hosted and macOS runners were not impacted by this incident. Overall, 9% of actions workflow runs experienced a start delay, with impact peaking at 21.4%. Among the affected runner categories, 78.99% of larger-hosted jobs, 29.8% of scale-set jobs, and 8.7% of self-hosted jobs took more than five minutes to acquire a runner. Reconnection traffic from affected runners also increased load on GitHub APIs, resulting in three to four seconds of additional average request latency and elevated 5xx error rates.
The incident was caused by a certificate lifecycle management failure in a subset of internal services, resulting in an SSL certificate expiration that disrupted runner connectivity. This specific certificate is used by minimum runner version enforcement. It relied on manual deployment and while a replacement was generated, it was not automatically deployed on receiving an automated alert. We restored service by rotating the affected certificate. Recovery began at 02:45 UTC. By 03:55 UTC, queued workflow backlog had been processed and workflow delay rates returned to normal.
GitHub has strengthened protections to limit the wider impact of runner connection issues. We are adding independent certificate expiration monitoring and improving renewal automation to prevent similar interruptions. We are also updating alerting, access, and response procedures so certificate concerns can be identified and addressed sooner.
July 21, 2026 (lasting 1 hour and 26 minutes)
On July 21, 2026, between 07:41 and 11:57 UTC, the SSH Authentication service on github.com was degraded, and some SSH connections using user RSA keys and deploy keys were rejected. On average, 12.2% of SSH authentication requests failed, peaking at 15.7%. Both user RSA keys and deploy keys were impacted.
This was due to a regression introduced during an unrelated internal infrastructure change. The regression affected a less common public-key authentication flow: a directly signed public-key method commonly used by deploy keys and automation. Valid authentication attempts using this method were rejected as invalid.
We mitigated the incident by reverting the change, after which SSH authentication returned to normal.
We are working to expand our automated test coverage for our SSH public-key authentication flows to catch more edge cases and to improve observability and alerting on SSH authentication failures, to reduce our time to detection and mitigation of issues like this one in the future.
July 24, 2026 (lasting 57 minutes)
On July 24, 2026, between 19:17 and 20:02 UTC, users were unable to create pull requests through web, command line, or API interfaces due to a database vschema change. In total, 113,930 pull request creation attempts were impacted across 50,904 users, with an average error rate of 1.75% and a maximum error rate of 2.25% for all requests to pull requests service. Existing pull requests and other GitHub functionality were not affected, although workflows dependent on creating new pull requests were also affected.
The root cause was related to a backfill workflow into the Vitess keyspace hosting pull request data. The backfill Vitess command encountered errors and increased VReplication lag, and the workflow was canceled at 19:17 UTC. The cancellation executed a misunderstood Vitess codepath that dropped the backing table to the target keyspace, leaving a non-existent reference that resulted in errors creating pull requests.
The issue was resolved by reverting the change to the affected database, upon which pull request creation immediately resumed. The mitigation was executing a command to drop the vschema reference to the dropped table, allowing pull request creation to resume.
Following the incident, we have added operational guidance for database index backfill and documented unexpected cancellation behavior. We are currently working on adding stronger pre-flight validation to our tooling, building capabilities into well known codepaths, and end-to-end testing in lower environments prior to production rollout.
July 25, 2026 (lasting 52 minutes)
On July 25, 2026, GitHub Actions experienced two related periods of degradation that caused some workflow runs to be delayed by more than five minutes or end with infrastructure failures.
The first degradation occurred between 08:45 and 09:13 UTC. During a planned availability-related configuration change on a critical-path Redis cluster for Actions, one participating region was left in a degraded state. Separately, an independent capacity operation temporarily removed another region from the cluster and redirected its traffic to the degraded region. This created cross-region inconsistencies in job-assignment state, causing workflow runs to be delayed, exhaust retries, or fail outright. At peak, about 7% of runs were delayed by more than 5 minutes, and 25% of runs failed with an infrastructure error during the course of the incident. We mitigated the incident at 09:13 UTC by returning traffic to its normal distribution.
Second period occurred between 12:08 and 12:48 UTC. As part of mitigating the first incident, traffic was returned to the regional instance that was still undergoing its capacity increase. Multiple Redis nodes in the scaling region experienced failures, increasing traffic to healthy nodes and causing connection limits to be reached on many nodes. At peak, 30% of runs were delayed by more than five minutes, and 60% of runs failed with an infrastructure error during the course of the incident. We mitigated the incident at 12:48 UTC by redirecting workflow traffic away from the scaling region.
We have added stronger regional health and capacity checks before maintenance, along with a required stable observation period before restoring traffic. We are continuing to improve automated connection resiliency and are partnering with our platform dependency to automatically detect and remediate unhealthy cluster members and shard imbalance. More broadly, we already had work underway to improve the resiliency and scale of this part of GitHub Actions infrastructure.
Follow our status page for real-time updates on status changes and post-incident recaps. To learn more about what we’re working on, check out the engineering section on the GitHub Blog.
The post GitHub availability report: July 2026 appeared first on The GitHub Blog.
Continue reading on the original blog to support the author
Read full articleThis report highlights the complexities of scaling a massive platform while migrating to cloud infrastructure. It provides insights into managing database hotspots, handling cascading failures in service meshes, and the critical trade-offs between availability and feature velocity.
This report provides a transparent look at large-scale infrastructure migration and service extraction. It highlights the trade-offs between speed and stability, the importance of edge rate-limiting, and how to handle database contention in a massive monolith-to-microservices transition.
This report highlights the challenges of scaling a massive monolith under AI-driven traffic growth. It provides a blueprint for reliability through infrastructure migration, service decomposition, and the implementation of automated circuit breakers to prevent cascading failures.
This architectural shift eliminates common failure modes in high-availability setups where search indexes could become locked or corrupted during upgrades. By using native Cross Cluster Replication, engineers gain a more resilient, easier-to-maintain search infrastructure.
