Salesforce Engineering
https://engineering.salesforce.com/Why it matters: For global-scale perimeter services, traditional sequential rollbacks are too slow. This architecture demonstrates how to achieve 10-minute global recovery through warm-standby blue-green deployments and synchronized autoscaling, ensuring high availability for trillions of requests.
- •Salesforce Edge manages a global perimeter platform handling 1.5 trillion monthly requests across 21+ points of presence.
- •Transitioned from sequential regional rollbacks taking up to 12 hours to a global blue-green model that recovers in 10 minutes.
- •Implemented parallel blue and green Kubernetes deployments to maintain a warm standby fleet capable of immediate full-load handling.
- •Customized Horizontal Pod Autoscalers (HPA) to ensure the inactive fleet scales identically to the active fleet, preventing capacity mismatches.
- •Automated traffic redirection using native Kubernetes labels and selectors instead of external L7 routing tools like Argo.
- •Integrated TCP connection draining and controlled traffic cutover to preserve four-nines availability during global rollback events.
Why it matters: This article details the architectural shift from fragmented point solutions to a unified AI stack. It provides a blueprint for solving data consistency and metadata scaling challenges, essential for engineers building reliable, real-time agentic systems at enterprise scale.
- •Salesforce unified its data, agent, and application layers into the Agentforce 360 stack to ensure consistent context and reasoning across all surfaces.
- •The platform uses Data 360 as a universal semantic model, harmonizing signals from streaming, batch, and zero-copy sources into a single plane of glass.
- •Engineers addressed metadata scaling by treating metadata as data, enabling efficient indexing and retrieval for massive entity volumes.
- •A harmonization metamodel defines mappings and transformations to generate canonical customer profiles from heterogeneous data sources.
- •The architecture centralizes freshness and ingest control to maintain identical answers across different AI agents and applications.
- •Real-time event correlation is optimized to update unified context immediately while balancing storage costs for large-scale personalization.
Why it matters: Securing AI agents at scale requires balancing rapid innovation with enterprise-grade protection. This architecture demonstrates how to manage 11M+ daily calls by decoupling security layers, ensuring multi-tenant reliability, and maintaining request integrity across distributed systems.
- •Salesforce's Developer Access team manages a secure access plane for Agentforce, handling over 11 million daily agent calls across production environments.
- •The architecture utilizes a layered access-control plane that separates authentication at the edge from authorization within the core platform to reduce latency and operational risk.
- •A middle-layer API service acts as a technical control point, ensuring all agentic traffic follows consistent security protocols and cannot bypass protection boundaries.
- •Security invariants include edge-level authentication validation, core-platform-enforced authorization, and end-to-end request integrity using Salesforce-minted tokens.
- •The system is designed to contain multi-tenant blast radius risks, preventing runaway agents or malformed requests from impacting other customers in a shared environment.
- •Strict egress traffic filtering and cross-boundary revalidation are employed to maintain the principle of least privilege across the distributed compute layer.
Why it matters: Benchmarking AI systems against live providers is expensive and noisy. This mock service provides a deterministic, cost-effective way to validate performance and reliability at scale, allowing engineers to iterate faster without financial friction or external latency fluctuations.
- •Salesforce developed an internal LLM mock service to simulate AI provider behavior, supporting benchmarks of over 24,000 requests per minute.
- •The service reduced annual token-based costs by over $500,000 by replacing live LLM dependencies during performance and regression testing.
- •Deterministic latency controls allow engineers to isolate internal code performance from external provider variability, ensuring repeatable results.
- •The mock layer enables rapid scale and failover benchmarking by simulating high-volume traffic and controlled outages without external infrastructure.
- •By providing a shared platform capability, the service accelerates development loops and improves confidence in performance signals.
Why it matters: Engineers must evolve recommendation engines from passive click-based tracking to active intent extraction. This shift enables autonomous agents to provide contextually relevant responses in real-time, solving the cold-start problem and handling unstructured data at enterprise scale.
- •Developed 'Understand User Intent' Agentforce action to transform unstructured conversational history into structured JSON intent signals using LLMs.
- •Re-architected personalization systems to prioritize real-time conversational intent over long-term behavioral history for higher relevance.
- •Implemented semantic catalog modeling to solve cold-start problems where historical engagement data is missing.
- •Integrated intent signals into existing Data 360 real-time ingestion pipelines to maintain low latency during agentic interactions.
- •Bridged language gaps by mapping user-specific terminology to standardized catalog metadata across multiple languages.
Why it matters: This migration provides a blueprint for modernizing stateful infrastructure at massive scale. It demonstrates how to achieve engine-level transitions without downtime or application changes while maintaining sub-millisecond performance and high availability.
- •Successfully migrated Marketing Cloud's caching layer from Memcached to Redis Cluster at 1.5M RPS with zero downtime.
- •Implemented a Dynamic Cache Router to enable percentage-based traffic shifts and double-writes for cache warm-up without application code changes.
- •Addressed functional parity risks by standardizing TTL semantics and key-handling behaviors across more than 50 distinct services.
- •Utilized service grouping by key ownership to prevent split-brain scenarios and data inconsistencies during the transition.
- •Maintained strict performance SLAs throughout the migration, sustaining P50 latency near 1ms and P99 latency around 20ms.
Why it matters: Scaling AI agents to enterprise levels requires moving beyond simple task assignment to robust orchestration. This architecture shows how to manage LLM rate limits and provider constraints using queues and dispatchers, ensuring reliability for high-volume, time-sensitive workflows.
- •Transitioned from a single-agent MVP to a dispatcher-orchestrated multi-agent architecture to support over 1 million monthly outreach actions.
- •Implemented persistent queuing to decouple task arrival from processing, creating a natural buffer for workload spikes and preventing retry storms.
- •Developed a constraint engine to enforce provider-specific quotas and LLM rate limits, ensuring compliance with Gmail and O365 delivery caps.
- •Utilized fairness algorithms like Round-Robin and priority-aware polling to prevent resource monopolization and ensure timely processing of urgent tasks.
- •Adopted a phased scaling strategy to evolve throughput from 15,000 to over 1 million messages monthly through parallel execution across 20 agents.
Why it matters: Automating incident response at hyperscale reduces human error and cognitive load during high-pressure events. By using AI agents to correlate billions of signals, teams can cut resolution times by up to 80%, shifting from reactive manual triage to proactive, explainable mitigation.
- •Salesforce developed the Incident Command Deputy (ICD) platform, a multi-agent system powered by Agentforce to automate incident response.
- •The system utilizes AI-based anomaly detection across metrics, logs, and traces to replace static thresholds and manual monitoring at hyperscale.
- •ICD unifies fragmented data from observability, CI/CD, and change management systems into a single reasoning surface for AI agents.
- •Agentforce-powered agents automate evidence collection and hypothesis generation, significantly reducing cognitive load for engineers during 3:00 AM incidents.
- •The platform has successfully reduced resolution time for common Severity 2 incidents by 70-80%, with many detected and resolved within ten minutes.
Why it matters: Scaling to 100,000+ tenants requires overcoming cloud provider networking limits. This migration demonstrates how to bypass AWS IP ceilings using prefix delegation and custom observability without downtime, ensuring infrastructure doesn't bottleneck hyperscale data growth.
- •Overcame the AWS Network Address Usage (NAU) hard limit of 250,000 IPs per VPC to support 1 million IPs for Data 360.
- •Implemented AWS prefix delegation, which assigns IP addresses in contiguous 16-address blocks to significantly increase network efficiency.
- •Navigated Hyperforce architectural constraints, including immutable subnet structures and strict security group rules, without altering VPC boundaries.
- •Developed custom observability tools to monitor IP fragmentation and contiguous block availability, filling gaps in native AWS and Hyperforce metrics.
- •Utilized AI-driven validation and phased rollouts to ensure zero-downtime migration for massive Spark-driven data processing workloads.
Why it matters: AI tools can boost code output by 30%, but this creates downstream bottlenecks in testing and review. This article shows how to scale quality gates and deployment safety alongside velocity, ensuring that increased speed doesn't compromise system reliability or engineer well-being.
- •Unified fragmented tooling across Java, .NET, and Python using a portfolio approach including Cursor, Windsurf, and Claude Code.
- •Achieved a 30% increase in code production with 85% weekly adoption of AI-assisted development tools among eligible engineers.
- •Mitigated senior engineer bottlenecks by implementing AI-assisted code reviews to handle routine checks and initial analysis.
- •Scaled quality gates by automating test coverage and validation workflows to keep pace with accelerated development cycles.
- •Integrated AIOps and telemetry analysis to maintain high availability and improve incident response across 25 Hyperforce regions.