For global-scale perimeter services, traditional sequential rollbacks are too slow. This architecture demonstrates how to achieve 10-minute global recovery through warm-standby blue-green deployments and synchronized autoscaling, ensuring high availability for trillions of requests.
By Sanjeev Chhabria, Aditya Kamath, Derek Donaldson, and Sandeep Siroya.
In our Engineering Energizers Q&A series, we highlight the engineering minds driving innovation across Salesforce. Today we spotlight Sanjeev Chhabria, Vice President of Engineering, who leads the Salesforce Edge (Trusted Perimeter Platform) team, building and operating global perimeter services that handle 1.5 trillion requests and approximately 23 petabytes of traffic monthly.
Explore how the team reduced global rollback time from eight to twelve hours to roughly ten minutes, re-architected Kubernetes deployments and autoscaling for true blue-green operation, and automated traffic cutover and TCP connection draining to preserve four-nines availability during worldwide rollback events.
Our purpose is to manage Salesforce Edge as the reliable entry point for Salesforce application traffic. We ensure consistent security, performance, and uptime through this system.
Salesforce Edge, our dependable perimeter platform with 21+ points of presence around the globe and expanding, resides directly within the data path for inbound requests for Salesforce Core and some non-Core services. It enforces security protocols, diminishes threats, and directs traffic dependably to downstream services. We deliberately manage Edge as a consolidated perimeter instead of numerous independent regional deployments. This strategy streamlines architecture, minimizes redundant logic, and decreases operational burden while enhancing uniformity across environments. Centralized enforcement fortifies blast-radius containment and facilitates consistent controls. Operating as a singular platform allows for quicker incident response without requiring region-by-region coordination.
Given that Edge is a global service that fronts a large amount of Salesforce traffic, it is imperative to avoid any unnecessary downtime. Any compromise directly affects customer access to Salesforce. This fundamental truth promotes a strong inclination towards automation, predictability, and swift recovery mechanisms integrated directly into the platform’s deployment and operation.
Salesforce Edge worldwide network as of January 2026.
The primary limitation was time. The current rollback method could not satisfy recovery needs. Individual Edge pods, running on EKS clusters, require several minutes to initiate, and pod creation is deliberately staggered to prevent overwhelming Kubernetes clusters and the Edge control plane. The deployment pipeline itself also introduces a delay before traffic reaches new pods.
Deployments were executed in regional phases for safety. While effective for roll-forward releases, this method became a weakness during rollback. Reverting a single region could take nearly an hour, and sequential rollback across the global fleet extended total recovery time to the eight-to-twelve-hour range.
Considering these limitations, reducing rollback time within the existing architecture was impractical. We required an approach where rollback did not necessitate rebuilding or restarting capacity. This led us to a blue-green model where the prior version already operates at full scale as a warm standby. Rollback then transforms into a controlled traffic redirection rather than a redeployment. This change reduced global rollback from eight to twelve hours to ten minutes.
SF Edge data path switch from blue to green clusters.
The above diagram depicts the blue and green edge clusters and a rollback taking place. The label (green or blue) in the Kubernetes deployment controls the active data path. In the diagram, we have decided to rollback from version v2 to v1, which involves modifying the color label to green in this case, which will cause traffic to start landing on the v1 version of the software.
Evaluating existing Kubernetes blue-green tools led us to Argo, an open-source Kubernetes deployment system. It is widely used for blue-green and progressive delivery workflows. While effective for many application workloads, its underlying assumptions do not apply to a global edge service.
Argo anticipates traffic switching at a layer-seven routing tier, but Salesforce Edge functions as the routing layer itself. It also lacks native support for maintaining two fully scaled global deployments simultaneously. Furthermore, it does not coordinate autoscaling behavior across blue and green fleets. Instantaneous cutovers at this scale can also disrupt load-balancing behavior.
These limitations meant that adopting Argo would have introduced unacceptable risk. Therefore, we implemented equivalent workflows directly on native Kubernetes constructs. We used existing deployment automation to achieve this. This approach provided precise control over deployments, scaling behavior, and traffic transitions without inheriting incompatible platform assumptions.
Deployment and rollback mechanics were established, making autoscaling the next critical limitation. Kubernetes does not offer a native blue-green deployment primitive, so we explicitly designed one. We introduced parallel blue and green deployments, ensuring only one receives production traffic at any given moment. The main challenge involved maintaining identical scaling for both deployments so either could immediately handle full load.
Autoscaling presented the most complex aspect. Our Horizontal Pod Autoscaler scales based on the average CPU utilization of the request-processing container. During blue-green operation, traffic might split between deployments. This would typically cause each to scale independently, resulting in insufficient capacity during cutover.
To resolve this, we adjusted the autoscaling calculation. It now evaluates CPU utilization across both blue and green deployments and scales based on the higher value. The same calculation is used by the blue and green horizontal pod autoscalers, resulting in the same number of desired pods for both deployments. This guaranteed capacity parity at all times. We also updated the deployment pipeline. It now determines the active color through service labels, deploys only the inactive color, and promotes it after validation by updating the service.
Traffic cutover faced limitations due to long-lived TCP connections. Even after updating Kubernetes service labels, existing connections stayed with their original pods until natural termination. At edge scale, these connections can last for hours, which is unacceptable during an emergency rollback.
To fix this, we implemented explicit connection draining. After redirecting traffic at the service level, we launch a Kubernetes job. This job securely connects to each pod using mutual TLS and issues TCP drain commands. This forces connections to close gracefully, so clients immediately reconnect to the correct deployment.
This method replaces unpredictable organic connection churn with controlled traffic migration. Most clients reconnect quickly. This allows traffic to converge on the rollback target within minutes and enables global rollback completion well within the recovery window.
With four-nines availability, the margin for error is extremely small. The entire annual downtime budget measures in minutes, not hours. Prolonged degradation from faulty releases can quickly exhaust that budget.
Before blue-green rollback, recovering from a bad release could consume a significant portion of the annual availability allowance in a single incident. Maintaining a fully scaled warm standby fundamentally changed that dynamic. Once a rollback decision happens, recovery becomes a rapid traffic redirection rather than a prolonged rebuild.
This capability has already proven critical. During a recent severity-one incident, we rolled back globally fast enough to restore service before escalation to a severity-zero outage. Automated rollback now acts as a safety net that materially limits customer impact when failures occur.
The post Inside Salesforce Edge: Automating Global Rollback for 1.5 Trillion Requests in 10 Minutes appeared first on Salesforce Engineering Blog.
Continue reading on the original blog to support the author
Read full articleSecuring AI agents at scale requires balancing rapid innovation with enterprise-grade protection. This architecture demonstrates how to manage 11M+ daily calls by decoupling security layers, ensuring multi-tenant reliability, and maintaining request integrity across distributed systems.
This system demonstrates how to transform massive, fragmented telemetry into actionable insights. By standardizing health metrics and isolating analytics from production, engineers can proactively identify risks, reduce support overhead, and ensure platform stability at a petabyte scale.
Automating large-scale infrastructure migrations is critical for reducing operational risk. MIPS demonstrates how to build a deterministic decision engine that maintains auditability and customer trust while scaling to handle tens of thousands of complex organization moves.
This migration provides a blueprint for modernizing stateful infrastructure at massive scale. It demonstrates how to achieve engine-level transitions without downtime or application changes while maintaining sub-millisecond performance and high availability.
