This article demonstrates how to scale an observability platform to 4B metrics/min while solving single-region dependencies. By moving to a geo-local model, Salesforce reduced blast radius and data transfer costs, ensuring global visibility even during regional infrastructure failures.
By Priyanka Singla, Aaron Stockton, and Vincent Poon
In our Engineering Energizers Q&A series, we highlight the engineering minds driving innovation across Salesforce. Today, we spotlight Priyanka Singla, Senior Software Engineer on our Argus team, who operates and evolves the internal observability platform ingesting approximately 4 billion metrics per minute for thousands of engineering teams.
Explore how the team scaled ingestion, storage, and query performance as telemetry volume grew across Salesforce systems, while also overcoming a single-region AWS dependency, whose blast radius could take down Argus visibility globally.
The team provides Salesforce with an enterprise-grade metrics platform for monitoring, dashboards, and operational visibility. Argus ingests telemetry from services across the company and stores that data at a massive scale. This makes information queryable for teams that depend on fast, reliable insight.
That mission extends beyond storage because the platform allows for simple onboarding. Teams adopt Argus without complex integrations or heavy setup. Once they finish onboarding, these teams gain dashboards, alerting, and metrics analysis through a common internal platform.
The team also operates Argus as a continuously evolving product. As telemetry patterns shift and new workloads emerge, the architecture expands while maintaining reliability. For thousands of internal users, Argus remains a primary source of metrics visibility across Salesforce services.
The primary scalability demand stems from the fact that observability growth rarely slows. As more services launch and systems become more distributed, telemetry becomes more granular and metric volume rises. Argus now handles roughly 4 billion metrics per minute, which creates significant load across ingestion, storage, and query serving layers.
At that throughput, ingestion is only the first constraint. The platform must also store and retrieve data efficiently: Argus retains telemetry for 45 days, while a separate store ingests the same data in parallel for longer retention. On the query side, performance is helped by metadata caching. When queries contain wildcards, previously resolved metadata is cached so subsequent queries matching the same pattern skip resolution and stay fast. Without that cache, every wildcard query would pay the full metadata-resolution cost and query latency would rise.
To keep pace, the team continuously re-architects Argus rather than relying on a fixed design. OpenTSDB and HBase serve as core storage layers while metadata caching keeps wildcard query resolution fast on the serving path. Horizontal scaling and targeted tuning across each layer ensure the system handles the volume because solving one limit usually means preparing for the next.
A look at the previous architecture.
For years, the biggest reliability risk involved operating Argus in a single AWS region. While the team solved most internal software issues directly, a regional infrastructure outage created a dependency outside its control. If that AWS region failed, Argus visibility could suffer as a result.
Beyond availability, the single-region model also drove up data transfer cost and latency, since telemetry from all other FIs/DCs/regions had to be shipped into this one central FI/region for processing.
Argus powers dashboards and metrics across Salesforce, so teams rely on it during incidents to isolate failures and coordinate recovery. The risk was not a regional outage itself, rather, it was that one region could take down
visibility globally, which is precisely the blast radius geo-local was designed to contain.
A regional incident in that one region could render the entire platform unavailable — its blast radius extended across all of Argus at once, with no isolation between what failed and what remained available. Providing high availability by keeping multiple copies of Argus data across the regions was expensive at Argus scale because the system already consumed substantial AWS resources. Reducing that blast radius required a redesign that was both technically and economically viable.
The transition to a multi-geo architecture required a major redesign rather than a simple deployment expansion. It was a six-month long effort that brought three teams together — Argus, Argus India, and the HBase team — spanning storage, and ingestion layers. The team required a model that improved resilience without making blast radius improvements too expensive.
Instead of full replication, Argus introduced a geo-local model that distributes data across multiple production geographies. Metrics undergo processing and storage closer to their origin, which reduces cross-region dependencies and improves fault isolation.
This shift required rethinking data placement, ingestion ownership, and regional responsibilities. The system also had to preserve existing functionality so internal dashboards and workflows continued without disruption. By distributing data intelligently rather than duplicating everything, the team created a practical path towards blast radius reduction while maintaining cost discipline.
This is now live across 4 of 5 production regions, with only the final region remaining, completing our production rollout.
A look at the latest architecture.
When data spans multiple geographies, read paths become more complex than in a centralized model. A previous request succeeded or failed against one region. Now, some geographies may respond while another remains unavailable. This creates an operating model that requires the correct handling of partial data.
The team introduced a federation query layer to address this challenge. This layer routes requests to regional query services, retrieves local results, and aggregates them for the caller. To route intelligently rather than broadcasting every query to every geography, the federation layer maintains metric scope to region-level metadata from all geographies, giving it a global view of where data actually lives. Backed by Elasticsearch mappings, this lets the federation layer resolve a query to only the relevant geographies and skip the rest. This avoids unnecessary cross-region fan-out, reduces load, and cuts latency on every request.
Latency also presented a concern. Cross-region hops introduce delays for near-real-time workloads. The team optimized performance so users gained resiliency without facing unacceptable response times.
The user experience evolved to match these technical shifts. Instead of silent failures, the system introduced partial-response semantics and HTTP 206 responses. UI signals now inform users when a geography becomes unavailable. This combination of federation, transparency, and graceful degradation makes multi-geo observability practical.
With geo-local now live in production across nearly all regions, the focus shifts from rollout to sustaining isolation as volume grows. As demand rises, isolation boundaries require constant maintenance to keep any single failure from widening its reach. Each year brings more services, distributed workloads, and telemetry sources that increase the load on ingestion and storage.
Recent AI adoption accelerates this trend. As AI workloads scale, existing instrumented code paths are exercised far more heavily, generating a much higher volume of signals. The observability platform must grow alongside the systems it monitors.
Keeping blast radius small requires continuous capacity planning and architecture reviews. The team identifies future bottlenecks in storage and network paths before users feel an impact.
This work requires disciplined operations. Multi-region systems reduce the blast radius of any single regional incident but introduce challenges around geo balancing and cross-region traffic costs. Sustaining fault isolation at this scale remains an ongoing engineering practice of constant adaptation.
The post How Salesforce Eliminated Single-Region Risk and Reduced Downtime Blast Radius at 4B Metrics/Min appeared first on Salesforce Engineering Blog.
Continue reading on the original blog to support the author
Read full articleThis article demonstrates how to build a resilient distributed system that handles extreme scale and unpredictable customer data models. It provides a blueprint for managing metadata bottlenecks and resource allocation when processing quadrillions of records across disparate storage systems.
Automating large-scale infrastructure migrations is critical for reducing operational risk. MIPS demonstrates how to build a deterministic decision engine that maintains auditability and customer trust while scaling to handle tens of thousands of complex organization moves.
This migration provides a blueprint for modernizing stateful infrastructure at massive scale. It demonstrates how to achieve engine-level transitions without downtime or application changes while maintaining sub-millisecond performance and high availability.
Scaling to 100,000+ tenants requires overcoming cloud provider networking limits. This migration demonstrates how to bypass AWS IP ceilings using prefix delegation and custom observability without downtime, ensuring infrastructure doesn't bottleneck hyperscale data growth.
