This article demonstrates how investing in in-house test infrastructure and smart sharding can drastically improve CI/CD efficiency and developer velocity by reducing build times and flakiness. It highlights the benefits of taking control over critical testing environments.
George Kandalaft | Software Engineer II, Test Tools; Alice Yang | Staff Software Engineer, Test Tools
Problem Statement
Our Android end-to-end testing builds in CI were slow and flaky because the test shards were unbalanced and the entire build had to wait for the single slowest shard on a third-party testing platform.
Solution
We built a runtime-aware sharding mechanism that uses historical test duration and stability data to pack tests greedily in an in-house testing platform with EC2-hosted emulators, ensuring all shards have a similar total runtime.
Impact
The end-to-end build time was reduced by 9 minutes (a 36% improvement), decreased the slowest shard’s runtime by 55%, and compressed the time difference between the fastest and slowest shards from 597 seconds to just 130 seconds.
At Pinterest, one of the Test Tools team missions is to save engineers time across development, testing, and shipping. Delivering on that mission requires fast, reliable feedback for our end-to-end (E2E) testing pipelines. As our app and test suites have grown, we faced a familiar challenge: how do we keep E2E tests fast and reliable as test volume and complexity rise in CI?
Our Android E2E tests run on emulators and devices that exercise real user flows. In CI, we traditionally split these suites into shards by package names. The test run and parallelism were managed by Firebase Test Lab (FTL) based on the packaged sharding. This type of sharding created imbalance: some shards finished quickly while others dragged on — so the slowest shard gated the build.
To make feedback predictable and fast, we moved to time-based sharding with a new in-house testing platform and infrastructure. Using historical runtimes, we assign tests so each shard takes roughly the same time, not merely the same count of tests. This post explains how we built in-house time-based sharding for Android at Pinterest, how we handle flaky tests, and how the change reduced build latency and improved developer velocity.
Key terms we’ll use
Before we could optimize our testing sharding, we needed to re-evaluate our testing execution environment. With FTL, there were some challenges. Each test run incurred a baseline setup time of five to six minutes. As our test suites grew and flaky tests required retries, this setup overhead became a significant portion (> 50%) of the total build duration.
Additionally, FTL infrastructure flakiness and instability started early 2023 blocked developer productivity one to two times per week on average, with each outage causing three to four hours of downtime and halting all code merges. There were no near term solutions provided by Google.
One of the first steps we took was to evaluate the vendor market for an alternative solution. The goal was to find a third-party solution that offers high reliability, low setup overhead, and massive parallelism on both emulators and physical devices. Emulator support is especially important as it offers much better cost and performance efficiencies for our large-scale test suite.
However, the vendor evaluation showed that the available solutions did not fully align with our requirements for large-scale, native emulator support. We found out that even with an alternative vendor, we would still need to build and maintain our own custom test orchestration and sharding mechanism. Below is a list of the critical acceptance criteria we had during the evaluation.
Based on the analysis, we concluded that building an in-house testing platform was the most direct path to solve the root cause of the problem to improve developer velocity and experience. Through conversations with peer companies who had successfully built their own Android testing platform, and an internal proof-of-concept, we’ve seen feasibility and promising performance gains.
This led to the creation of PinTestLab, Pinterest’s in-house Android E2E testing solution that runs on EC2-hosted Emulators. This gave us direct control over the entire testing stack, from docker images, emulator configurations, error logging, and sharding.
A smart sharding algorithm is only as effective as the infrastructure it runs on. Building PinTestLab required a stable and efficient foundation. This involves a solid proof-of-concept (POC) to identify the right instance type, followed by iterative optimization to maximize resource utilization.
Our initial POC explored running Android emulators on standard, virtualized EC2 instances (x86 architecture) inside a docker image. It revealed significant performance bottlenecks. A switch to bare-metal instances immediately resolved these issues, as shown by the drastic improvements in emulator boot time and test time.
We also briefly investigated using ARM-base bare-metal instances but encountered significant tooling friction and kernel panics during emulation.
Based on these findings, we selected c7i.metal-24xlarge instances for all test executions. Our initial rollout established a baseline, which we later improved through resource allocation optimization to our docker image. This enhancement, combined with the improved load distribution from time-based sharding, allowed us to better utilize host resources and increase peak memory usage to 80%.
In December 2024 we launched PinTestLab minimum viable product (MVP). We wanted to cut external flakiness and regain control over device images, scheduling, and retries — while keeping engineers’ CI/CD flow in Buildkite.
For the MVP, we kept orchestration simple. Buildkite YAML defined the end‑to‑end test run. A Python orchestrator spun up one emulator per shard, ran a pre‑assigned test list, uploaded artifacts, and then shut the emulator down. Sharding used a uniform round‑robin split by test count, with package‑based splits as a baseline. We recorded per‑shard timings and outcomes as Buildkite artifacts and sent events to an observability platform to spot skew and outliers, and we added conservative shard‑level retries. This gave us stable runs under our control.
In FTL we split tests by package: all tests within a single package were grouped together and put into one shard. This was slow, and some packages containing long running or resource intensive tests took too long to finish, increasing total build time. We decided to move away from this toward a more balanced approach.
The first version kept things simple: scan the repo, divide the same total number of tests into shards round‑robin by count, and generate Buildkite steps to execute them. It worked, but shard durations were uneven: across 17 shards we saw P90 runtimes from roughly 8.8 to 13 minutes, which meant one long shard often gated the entire build. Our goal was to tighten that spread so that the “tail” stopped owning the overall time.
Two practical constraints shape how we do this:
That’s why equal test counts per shard can still be slow. In the example, Shard 1 bunches three tests on one emulator while two emulators sit idle, so the shard’s wall‑time is dragged out by that single busy emulator and any retry there makes it worse. Shard 2 spreads work across emulators, so it finishes sooner even with the same number of tests.
For PinTestLab, we kept orchestration simple at first, so we split tests with a round‑robin approach. It was better than the old package‑based splits, but still not ideal. This six test example made it obvious.
You get Shard_1 = [test_1, test_3, test_5] and Shard_2 = [test_2, test_4, test_6]. Inside the shard, the scheduler runs tests on two emulators; Shard_1 ends up with one emulator running for 10=6+4 minutes, while the other finishes a single 3 minute test and sits idle. Shard_1 takes 10 minutes, and Shard_2 takes 4 minutes. Even though we “balanced” by test count and total time looks fine on paper, one emulator in one shard becomes the straggler, causing a total build time of at least 10 minutes.
Even with equal test counts per shard, one emulator in shard_1 becomes the straggler, so total build time is ≥ 10 minutes. Round‑robin balances lists.
Attempt 2: Sort, then round‑robin
Sorted tests by expected time: [6, 4, 3, 2, 2, 2] → [test_1, test_5, test_3, test_2, test_4, test_6] Shards (round‑robin after sort):
Sorting reduces the skew (8 vs. 10 minutes) but still leaves idle time on the second emulator in each shard and a longer shard that gates the build.
We decided to develop a runtime-aware sharding, and we had a promising first run. The numbers speak for themselves. Under round-robin sharding, our slowest shard took 522 seconds, while the fastest finished in just 246 seconds. A 276-second spread meant developers waited for the tail every time.
With time-based sharding using p90 duration as our metric, we compressed that spread to just 28 seconds (5.6 vs 5.1 minutes). In practice, this meant cutting typical CI feedback from nearly 9 minutes down to about 5.5 minutes; this is a significant boost to developer velocity.
To address the timing imbalance, we leveraged historical runtime data stored in Metro, Pinterest’s test management system. Metro tracks test results, trends, ownership, and build history, while allowing engineers to debug test failures and take actions, such as quarantine flaky tests when necessary.
Using this rich historical data, we developed a greedy, runtime-aware allocation algorithm. The approach is straightforward: we sort tests by a robust performance metric (such as the average duration), add a small per-test overhead to account for setup costs, then pack tests into shards using a greedy strategy. Each test gets assigned to whichever emulator is projected to finish earliest, keeping all execution emulators busy and ensuring emulators within each shard complete at roughly the same time. This approach reduces tail latency without adding complexity to our CI orchestration.
Our initial implementation created a minimal runtime-aware sorter that ran entirely within Buildkite. We added a step that fetched historical data from Metro for the current test suite, calculated average runtimes based on the past x runs from the pipeline, then used a min-heap data structure to continuously assign the next test to the shard expected to finish first.
We schedule tests using a simple, fast heuristic. Let N be the number of tests and M be the number of emulators. We first sort tests by descending average runtime, then assign each test to the currently least loaded emulator using a min-heap. This is the classic Longest Processing Time (LPT) rule for identical machines: it isn’t guaranteed to be optimal, but it’s well suited to our CI. Computing the exact optimum is NP-hard and impractical at our scale, while LPT runs in O(N log N + N log M), scales cleanly across shards×emulators, and delivers near‑optimal finish times in practice. Given runtime estimates and the need for predictable, low‑overhead scheduling, we favor effectiveness, simplicity, and operational reliability over theoretical optimality.
Why this works well for us:
How the algorithm works (step-by-step):
Why is this?
To scale beyond post-commit, we decoupled our sharding algorithm from the pipeline. Because we already had rich historical data in the Metro, we built the solution there to be reusable for any testing pipelines. Given a list of tests plus the desired number of shards and emulators, the system computes an optimal sharding plan and estimates the fastest and slowest shard, as well as the fastest and slowest emulator if applicable. It also tracks prediction accuracy over time, so we can both forecast performance and measure how well those forecasts hold up.
ALGORITHM: Runtime-Aware Test Allocation
INPUT:
- tests: List of test cases
- num_shards: Number of available shards
- num_of_emulators: Number of available emulators
- per_test_overhead: Setup cost per test (constant)
OUTPUT:
- shard_assignments: Map of shard_id -> List of tests
This allowed us to use Metro as a brain given that we already have the tests duration in Metro. In addition, it also provided us with a rough estimate for the slowest and fastest shard giving us insights on the overall run time of the build.
We also explored a second path: on‑demand sharding driven by a message queue. This would let emulators pull work dynamically rather than running a precomputed test list. It’s promising, but it requires a separate proof of concept with third party dependencies evaluation.
We introduce an SQS layer to dispatch individual tests and create logical queues keyed by build_id and device_type so each build and device family is isolated.
Each shard spins up N emulators; every emulator becomes an SQS consumer. Instead of a static list, emulators fetch test messages on demand. For each message, the consumer asks Metro whether the test is already done (passed) or has failed out by hitting the retry limit. If neither, the emulator runs the test. On success, the consumer deletes the message from SQS. On failure, it re‑enqueues the test with retry metadata (attempt count, last error, backoff hints) so another consumer can pick it up later.
This push‑pull model keeps emulators busy and naturally shifts capacity toward longer or flaky tests during the run.
We chose time‑based sharding for the best ROI at the moment, as it already serves the purpose and meets (and exceeds) our expectations: tightening tail latency with minimal orchestration changes. It offers lower uncertainty and an existing prototype, reducing implementation risk and operational overhead. On‑demand sharding remains a viable next step: if our test mix becomes more skewed, or we need finer‑grained elasticity and targeted retries at very large scale, we can integrate Metro with SQS and pilot dynamic dispatch behind a feature flag. For now, runtime‑aware sharding provides predictable, fast feedback with a smaller blast radius and simpler operations.
To measure impact, we ran two builds in parallel to capture up‑to‑date results. Compared with sorted count‑based sharding, runtime‑aware sharding reduced the slowest shard from 863s to 392s (−54.6%), lowered average shard time from 400.1s to 303s (−24.3%), and cut total build time by about nine minutes (−36%). Bottom line: runtime‑aware sharding shortens end‑to‑end build time by roughly nine minutes while cutting tail latency by about 55%.
Kudos to everyone involved! Among them:
Building a fast and reliable CI/CD pipeline is a continuous journey. What challenges have you faced with test parallelization in your organization? Share your thoughts or questions in the comments below!
Slashing CI Wait Times: How Pinterest Cut Android Testing Build Times by 36%+ was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Continue reading on the original blog to support the author
Read full articleThis article demonstrates how to overcome legacy observability challenges by pragmatically integrating AI agents and context engineering, offering a blueprint for unifying fragmented data without costly overhauls.
This article details Pinterest's approach to building a scalable data processing platform on EKS, covering deployment and critical logging infrastructure. It offers insights into managing large-scale data systems and ensuring observability in cloud-native environments.
This article highlights the extreme difficulty of debugging elusive, high-impact performance issues in complex distributed systems during migration. It showcases the systematic troubleshooting required to uncover subtle interactions between applications and their underlying infrastructure.
This article details Pinterest's strategic move from Hadoop to Kubernetes for data processing at scale. It offers valuable insights into the challenges and benefits of modernizing big data infrastructure, providing a blueprint for other organizations facing similar migration decisions.
