This report provides a transparent look at large-scale infrastructure migration and service extraction. It highlights the trade-offs between speed and stability, the importance of edge rate-limiting, and how to handle database contention in a massive monolith-to-microservices transition.
Last month, we added a new section to our monthly reports with updates on GitHub’s availability work and the infrastructure investments behind it. The response and the questions we heard made one thing clear: customers want more of this, not less—including when the news is mixed.
The short version of June: We made real structural progress, we paused deliberately when a ramp went sideways, and we missed a target that we’ve now re-baselined.
Monolith traffic in Azure peaked at 45% in Central US this month. That number is lower than we’d hoped, because we paused the ramp for roughly a month after a stability incident on May 21 made it clear the environment wasn’t ready for more traffic. We restarted the ramp on June 17 with a new per-turnup stability gate that requires the environment to be verifiably healthy before each step up. Going more slowly with more confidence is the right trade—a controlled pause is preferable to relearning the same lessons at higher load.
Git in Azure grew from 30% to a peak of 43% (HTTP and SSH combined) over the month, and we missed our June target of 50%. We expect Git to plateau near 45% for now because of two deliberate decisions to avoid added user latency: we are waiting on additional vPoP traffic to route to Central US rather than backhauling IAD HUB Git traffic, and we are routing only HTTP for now because SSH has no read/write split at the edge. We continue to prioritize this, but we don’t have a new target to share. We’ll keep working as quickly and safely as possible and will report the specific numbers in our next update.
Underneath those headline numbers, we made progress that we are particularly excited about. Our new extracted pull requests service, pullsd, is now handling 100% of anonymous pull request reads in production; this traffic is no longer served by the monolith. Reposd, our new extracted repository service, became the first extracted service to serve production REST traffic from Azure, ramping to 50% of read traffic before we proactively turned it down for a Redis capacity constraint. There was no incident and no rollback under duress. It will re-ramp once that capacity work completes. Our new users service is now offloading roughly 500,000 queries per second at peak from our primary database, with the physical migration of authentication and authorization tables landing in early July. Our API rate limiting is now approximately 97% handled at the Gateway, so rate-limiting decisions no longer contend with request-serving workers inside the monolith. Client-side database load shedding is running against 5% of real production traffic, which means we now have live evidence that we can shed low-priority queries under stress before they cascade into user-facing failures. And two-person confirmation is now required end-to-end for interactive production access and ChatOps changes, with a unified audit trail behind it.
The incident write-ups that follow are the other half of the picture: what the system did well and what it didn’t, what we’ve already changed as a result, and what we’re still changing. The same principle continues to guide us: availability, then capacity, then features.
In June, we experienced six incidents that resulted in degraded performance across GitHub services.
June 04 17:30 UTC (lasting 1 hour and 25 minutes)
On June 4, 2026, from 17:30 to 18:55 UTC, Copilot code review experienced elevated failures for review requests on github.com. Affected users saw “Copilot ran into an error” on pull requests when requesting a code review.
During the incident window, an average of 81.6% of Copilot code review requests failed, with a peak failure rate of 93.9%, and a total of approximately 36,800 code review requests failing. GitHub Enterprise Cloud with data residency was not impacted.
The issue was caused by a newly released dependency used by the Copilot code review processing workflow. The release introduced an incompatibility with the runtime environment. Because the workflow automatically consumed the latest release, the incompatible version was picked up without sufficient compatibility validation and caused review processing to fail. Affected review jobs did not fail fast; many continued running until they timed out.
We mitigated the incident by removing the problematic dependency version and redeploying the affected processing service. New code reviews began recovering at 18:44 UTC, and the failure rate returned to baseline by 18:55 UTC. Remaining timed-out work drained by 19:59 UTC.
To reduce the risk of recurrence, we are pinning the dependency version instead of automatically consuming the latest release, adding compatibility checks for future releases, improving fast-failure behavior when the review processor cannot start, adding shorter timeout controls for review workflows, and improving monitoring for review completion failures.
June 08 06:30 UTC (lasting 2 hour and 06 minutes)
On June 8, 2026, between approximately 06:30 and 08:36 UTC, signed-out users experienced sustained elevated HTTP 504 errors when accessing pull requests, issues, releases, patch diffs, and other related github.com pages. During the incident, approximately 17% of unauthenticated requests to the affected github.com endpoints returned gateway timeout errors, peaking at roughly 34% of requests at around 06:50 UTC. Some GitHub Actions workflows were also affected when they depended on release downloads or related github.com endpoints. The impact lasted approximately two hours and was isolated to unauthenticated traffic; signed-in users were not affected.
The issue was caused by a significant increase in abusive, automated anonymous traffic to specific github.com endpoints. Because unauthenticated requests are served by a dedicated pool of web application servers, it degraded our ability to respond to unauthenticated requests, causing requests to queue beyond timeout thresholds and return gateway timeout errors.
We mitigated the incident by identifying the anomalous traffic pattern and applying targeted blocks at the load balancer and application layers. Once the blocks took full effect, error rates returned to normal and affected services were fully restored by 08:36 UTC.
To reduce the likelihood and impact of similar incidents in the future, we are improving automated detection and blocking for these traffic patterns, improving our emergency traffic-blocking deployment path, and evaluating routing changes for endpoints used by both signed-out users and automated workflows.
June 10 15:05 UTC (lasting 1 hour and 20 minutes)
On June 10, 2026, between 15:05 and 16:25 UTC, GitHub API services experienced degraded availability due to sporadic authentication failures affecting approximately 9% of requests. Both REST and GraphQL API requests were affected. Customers experienced intermittent “logged out” behavior as erroneous 401 (unauthorized) responses caused first- and third-party app integrations to trigger repeated authentication flows. Because only requests routed through the affected infrastructure failed, the same client could succeed on one request and fail on the next, producing the intermittent behavior. Affected requests also experienced approximately 800ms of additional latency, as the gateway retried authentication before returning an error.
A memcached proxy service, rollout to our internal API infrastructure caused our authentication service to pick up an incorrect host configuration, leading to intermittent authentication lookup failures. We mitigated the incident by deploying a configuration change to memcached service to use the correct host.
To prevent similar issues in the future, we plan to migrate our authentication system to the new caching infrastructure to improve resilience and strengthen overall reliability posture. We are also improving how the gateway distinguishes transient authentication-system errors from genuinely invalid credentials, so that temporary lookup failures no longer appear to users as being logged out.
June 16 17:20 UTC (lasting 55 minutes)
On June 16, 2026, between 17:20 and 18:15 UTC, the Opus 4.8 model experienced degraded availability in GitHub Copilot. During this window, some requests to Opus 4.8 failed or errored. Other Copilot models were not affected and remained available as alternatives. This was caused by an issue with an upstream model provider.
While the issue was ongoing, we enabled degraded-mode messaging to inform affected users. The upstream provider resolved the issue, and we monitored Opus 4.8 until success rates returned to normal. The incident is fully resolved.
We are reducing our reliance on any single inference provider for a given model and balancing capacity across providers, so traffic can fail over to healthy capacity during an upstream outage. Separately, we hardened our public status-page tooling to ensure incident updates publish reliably, improving how we keep customers informed during mitigation.
June 17 03:50 UTC (lasting 54 minutes)
On June 17, 2026, between approximately 03:50 and 04:44 UTC, GitHub Copilot was degraded and most of its frontier chat models were temporarily unavailable across all regions. During this window, affected models either disappeared from the model picker in the web, editor, and CLI experiences, or returned a “model not available” error when selected. Customers could continue using GitHub Copilot by selecting one of the models that remained available. The incident occurred during off-peak hours, which limited the number of customers affected.
This was due to a configuration change that our production system deemed invalid. We mitigated the incident by reverting the configuration change, after which the affected models returned automatically as the service reloaded the previous configuration.
We are working to roll out configuration changes gradually with stronger validations, alerts on sudden drops in the number of available models, and automatically roll back configuration changes that trigger these alerts.
June 25 17:33 UTC (lasting 23 minutes)
On June 25, 2026, between 17:33 and 17:55 UTC, our background job service experienced degradation which increased delays to pull requests, repository pushes, actions workflows, and webhooks, with delays peaking at 7 minutes. The issue was caused by underlying hypervisor issues and an incoming traffic spike, causing service timeouts which led to a connection storm and continual rebalances.
The issue was mitigated by replacing the impacted node at 17:49, after which all services saw recovery by 18:07.
To reduce the likelihood and impact of similar incidents in the future, we have made background job processing more resilient to sudden traffic spikes, reduced the possibility of connection churn in degraded cases, removed the co-location of critical nodes so that one unhealthy node cannot affect others, and added earlier alerting on the conditions that led to this degradation.
Follow our status page for real-time updates on status changes and post-incident recaps. To learn more about what we’re working on, check out the engineering section on the GitHub Blog.
The post GitHub availability report: June 2026 appeared first on The GitHub Blog.
Continue reading on the original blog to support the author
Read full articleThis report highlights the challenges of scaling a massive monolith under AI-driven traffic growth. It provides a blueprint for reliability through infrastructure migration, service decomposition, and the implementation of automated circuit breakers to prevent cascading failures.
This architectural shift eliminates common failure modes in high-availability setups where search indexes could become locked or corrupted during upgrades. By using native Cross Cluster Replication, engineers gain a more resilient, easier-to-maintain search infrastructure.
Git 2.55 brings critical performance enhancements for large-scale repositories. By optimizing multi-pack indexes and connectivity checks, it significantly reduces maintenance overhead and push latency, which is vital for SREs and developers managing massive codebases.
This report highlights the complexity of maintaining high availability in distributed systems. It provides lessons on the risks of automated infrastructure changes, the importance of correctly scoped rate limiting, and the need for robust DNS management and failover strategies.
