This article demonstrates how to overcome the 'scale wall' in big data by shifting from sequential to parallel experimentation. It highlights the importance of infrastructure that supports isolated, concurrent testing and identifies specific Spark memory pitfalls like object boxing.
By Prashanth Gedde Narayanaswamy
As one of Netflix’s largest and critical data pipelines, member-sessionizer consolidates tens of billions of hourly client events, encompassing user taps, playbacks, and engagement signals, into individual viewing sessions per profile.
A quick two-week tuning effort spiraled into two months when standard approaches hit a wall. Because the failures only happened at full production scale, each test took hours, limiting me to just three or four attempts daily. The turning point was changing the goal: instead of trying to speed up a single run, I built a way to run and monitor multiple tests at once and track every result.
That mindset led to an unconventional setup: kicking off five Spark jobs at once and monitoring them over spotty inflight Wi-Fi during a day-long trip home, while cloud servers handled the actual heavy lifting. This post covers how I set up that loop, along with the Spark bottlenecks and memory quirks I uncovered along the way.
Standard engineering wisdom says that you should isolate bugs on small datasets to iterate quickly. That works for standard software bugs, but it fails completely for big data performance bottlenecks.
While upgrading our sessionization pipeline from a legacy logging architecture to a modern platform, the new job hit critical failures, including executor Out-Of-Memory (OOM) errors, severe shuffle spillage, and thousands of stage retries. None of these issues could be reproduced using sampled data. The failures only triggered at full production scale because of data skew: an idle Netflix tab left streaming overnight generates tens of thousands of telemetry events, while a typical session only produces a handful. A standard 1% sample misses these extreme long-tail sessions entirely.
To catch the real bugs, every test run had to execute against a full hour of unsampled production data, along with all the deployment overhead that came with it.
The biggest win came from a feature in Dataflow, Netflix’s command-line tool for building, testing, and deploying data pipelines. It supports branch-driven development: our workflows run on Maestro (Netflix’s workflow orchestrator), which schedules and executes the jobs, and they already use a sanitized branch name (the ${dataflow.sanitizedbranch} placeholder). So if I push a branch like pgn/cap-experiment, then the deployed workflow, its tables, and its run namespace all get tagged with that branch name. This eliminates collisions with team members or between my own concurrent runs.
With that isolation in place, the model was simple: one branch is one independent experiment. I parameterized the scheduler (cap size, shuffle partitions, heavy-device threshold, feature flags) and gave each variant its own branch and output table. Instead of queuing five sequential tests over 10 hours, I fired off five variants simultaneously against the same time window. Total execution time dropped from the sum of all runs to just the length of the single slowest job.
Managing per-branch tables added zero overhead. Dataflow provisions and updates tables automatically from declared schemas without manual DDL. Whenever I modified the event bag structure between experiments, the table schema updated seamlessly, no CREATE TABLE or ALTER TABLE required.
The only real constraint was cluster contention. Because these heavy Spark jobs shared resources with the rest of the company, peak hours sometimes limited me to two concurrent runs instead of five. But even running two or three jobs in parallel was a massive improvement over single-threaded debugging.
The key takeaway: don’t focus on making a single multi-hour run faster, build a setup where you can run many experiments simultaneously without them stepping on each other.
While the previous chart shows the time savings, the architecture that enables this isolation looks like below:
Running five parallel experiments gets messy fast. Tracking instance IDs, configs, memory spills, and stage failures across variants makes it easy to lose track.
This is where an AI agent stepped in, not to write clever code, but to handle operational grunt work. I described variants in plain English, and the agent updated scheduler configs, deployed to the branch, triggered jobs, and polled metrics. Once complete, it compared spill and failure counts across variants in a summary table, allowing me to ask, “Which run is closest to OOM-free at peak load?” and get an immediate answer.
The value wasn’t deep Spark expertise, but operational memory. It turned tedious Spark UI cross-checks into a single prompt.
Eventually, I packaged this workflow into a reusable agent skill (.agents/skills/sandbox-deploy-and-run/) run via /sandbox-deploy-and-run <module> <scheduler-file> [PARAM=value]. It handles the full lifecycle—preflight, deploy, background polling, auto-diagnostics, and re-entrant resume <instance_id> support—turning multi-hour experiments into a single command.
All of this ran on Workbench, Netflix’s cloud dev environment, using my laptop as a thin terminal. Without session persistence, an SSH drop or sleeping laptop would kill multi-hour Spark runs mid-flight.
tmux running directly on Workbench solved this. Every experiment lived in a named cloud session: I could close my laptop, reattach hours later, and pick up where I left off. Combined with branch isolation, this made "start five and walk away" possible.
This is now my default setup: a local terminal (Ghostty), dedicated git worktrees per branch, and persistent tmux sessions on Workbench. The local laptop is disposable—closing it or losing network never interrupts the jobs.
That setup turned a long travel day into active iteration time. I kicked off batches at the gate, checked results via AI agent over spotty inflight Wi-Fi, and squeezed 4–5 full experiment cycles into a single 24-hour flight.
The root cause came down to a single operator: collect_list. Because Spark buffers each full event bag in memory, long-session OOMs were driven by individual bag size, not total dataset volume. The legacy pipeline handled identical row counts without crashing, proving that each event in the new bag had grown significantly heavier.
Two factors inflated event weight:
The Fixes:
Takeaway: Reducing individual event weight delivered far better stability than throwing more cluster memory at the problem.
Out of dozens of runs, two findings completely overturned my assumptions about Spark memory management:
The Surgical Fix: Instead of disabling broadcast joins globally, I lowered only AQE’s runtime threshold (spark.sql.adaptive.autoBroadcastJoinThreshold) to 10 MB while leaving the compile-time threshold untouched. That single change freed up ~3.1 GB of executor memory without impacting initial query planning.
Takeaway: Broadcast joins speed up queries, but pinned memory isn’t free. Watch accumulated broadcasts in your execution plan, not just dataset sizes.
When running dozens of parallel tests, the biggest risk isn’t job failure, it’s losing track of what you’ve already tried or confusing which JAR and config produced a result.
To solve this, I scaffolded the AI agent to append a row to a repository Markdown table after every run:
If a run yielded a key finding, the agent added a short note explaining what the result proved or disproved.
Key Takeaways:
A secondary doc (events-join-spark-tuning.md) captured the distilled findings as a quick "symptom-to-knob" lookup, so teammates didn't have to wade through the full chronological history.
To cap session size, I added row_number().over(Window...) and filtered for the first N events. It passed off-peak tests cleanly, but crashed with OOMs at peak load.
The issue was structural: Spark’s Window operator buffers an entire session in memory before evaluating row numbers and filtering. The cap trimmed output downstream, but it didn't limit memory usage during buffering, so a single skewed session overwhelmed executor memory.
The Fix & Lesson: We moved truncation ahead of the Window stage to cap events before buffering occurs.
Takeaway: Test fixes against worst-case input skew. Off-peak runs hide structural bottlenecks that peak scale will immediately trigger.
Pulling it together, four changes shipped, each isolated to a single experiment before it landed:
While these fixes made the pipeline run cleanly, they didn’t solve the underlying scaling issue: aggregating an entire session’s event bag into a single row has a hard ceiling. As our upstream telemetry source evolves and adds new attributes, each event in the bag becomes heavier, meaning this memory pressure will eventually return without a single line of code changing on our end.
Slimming and unboxing bought us headroom, but they didn’t change the fundamental data shape. The biggest takeaway from this effort is that tuning buys time, not infinite scale. If your underlying data model allows unbounded row growth, tuning alone won’t save you , eventually, the architecture itself has to change.
Running Apache Spark experiments in my sleep (and on a plane) was originally published in Netflix TechBlog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Continue reading on the original blog to support the author
Read full articleAt Netflix scale, raw telemetry is insufficient without context. This ontology-driven approach bridges disparate tools into a machine-readable knowledge graph, enabling faster root cause analysis and providing the structured context necessary for reliable AI-driven operations.
This article provides a practical blueprint for integrating large-scale foundation models into production. It highlights the trade-offs between embeddings, subgraphs, and fine-tuning, helping engineers balance performance, latency, and infrastructure costs in ML pipelines.
DataJunction solves 'metric sprawl' by providing a centralized semantic layer. It allows engineers to define metrics once and use them everywhere, ensuring data consistency and reducing onboarding time from weeks to minutes in complex environments like experimentation platforms.
Managing petabyte-scale time-series data in hot storage is prohibitively expensive. This tiered approach maintains low-latency access for recent data while leveraging cheap object storage for historical audits, significantly reducing costs without breaking the unified query API.
