Traditional testing fails with non-deterministic LLMs. Eval-driven development provides a systematic framework to ensure GenAI reliability, preventing regressions and aligning model outputs with product goals through structured, multi-layered validation.
How Airbnb teams build trustworthy Generative AI products by treating evaluation as a first-class engineering discipline; not an afterthought.
By: Rohit Girme, Dan Miller, Mia Zhao, Lifan Yang, Clint Kelly
Generative AI breaks a lot of the assumptions that used to hold true for software testing. Unlike traditional software, LLM outputs are non-deterministic, and “correct” is subjective. Because so much judgment is involved, you often need an AI to evaluate an AI, which introduces its own potential failure modes. Making matters more complicated, a single interaction with an LLM can chain retrieval, reasoning, tool calls, and generation, each of which can fail independently.
At Airbnb, we build LLM-powered features across our product, with recent launches including review highlights, AI customer support, smart communication features for guests and hosts, and more. Behind the scenes, we also use AI to help us spot trends and understand what’s working, guiding where we improve the product next.
Each product team may have its own evaluation criteria, process, workflows, etc. However, these are built on top of some common foundations and principles. An infrastructure team provides tooling and best practices, incorporating learnings across domains so that they are shared with everyone building products at Airbnb.
In this article, we wanted to share some of these best practices and learnings with the broader engineering community. Please note that the recommendations here are not intended to be prescriptive; there is no one-size-fits all approach when it comes to running evals.
Evaluating LLM-based systems is challenging work, and this should be planned for at the outset. Without a deliberate strategy, three things tend to happen:
Expect to spend a meaningful share of your total project effort on evaluation. This is not unnecessary overhead, it’s how you build products that actually work.
When in doubt, look at your data. Manually reviewing your data and building an intuition for what counts as success is always the starting point we recommend to teams. Build your prototype, and run it through 100 examples (synthetic is fine). Then read the outputs. Read the traces and find the model’s mistakes. Categorize them and build an eval.
This single habit will do more for your product quality than any framework, tool, or methodology in this document.
Formalized, that habit becomes eval-driven development (EDD), the GenAI analogue of test-driven development. Rather than predicting every failure upfront, EDD builds the infrastructure and habits to discover, encode, and continuously test for failure modes as they appear. It also forces stakeholders to externalize what “good” means, which shapes the product roadmap.
Five principles anchor EDD:
Every evaluation you run will use one or a combination of these three methods.
Layer 1: Programmatic checks (fast, low resource — catches obvious failures)
↓
Layer 2: LLM-as-a-Judge (nuanced - catches quality issues)
↓
Layer 3: Human evaluation (high resource - validates edge cases,
calibrates the stack)
Deterministic, code-based checks that don’t require an LLM call should be your first filter, catching obvious failures before you send anything to a judge or human labeler.
✅ Do: Use structured outputs (JSON schemas) to ensure strict typing.
❌ Don’t: Rely on prompt instructions alone to format data. This breaks downstream data pipelines.
Use a stronger LLM to evaluate another LLM’s output against a carefully designed rubric. This is how you assess nuanced qualities e.g. tone, coherence, faithfulness, relevance, at a fraction of the resources needed for human evaluation.
Rubric design matters. Ambiguity is the enemy. Something like “Is the provided explanation readable and up to our standards?” isn’t likely to be effective — if a human can’t apply the rubric consistently, an LLM certainly can’t.
Here is a simplified example of a single virtual judge’s rubric:
Score the readability of listing explanations. A good explanation sounds
like a friendly travel agent: warm but professional,
simple, natural, grammatically complete.
Score 1 if it reads cleanly.
Score 0 if it has ANY of these problems:
- Tone: too formal/jargony, too casual
("awesome vibes"), too salesy ("amazing!"), or robotic.
- Internal terms: never use internal terminology.
- Formatting: no quotation marks, no bullets, no fragments. End every
explanation with a period - never "!" or "?".
- Grammar: use articles/determiners/prepositions for natural flow
("this home has a pool", "close to downtown"). In a series, use the
article once then drop it: "a backyard, grill, and kitchen" - not
repeated, not omitted entirely.
- Complexity: plain words over jargon ("pool" not "aquatic recreation
area"; "near" not "proximate").
Examples:
- "Host mentions a pool and hot tub available near downtown." → 1
- "The listing mentions a pool!" → 0 (internal term "listing"; ends in "!")
- "This domicile encompasses aquatic amenities." → 0 (complex words; jargon)
Return ONLY:
{
"reason": "<list of [error_type, explanation] tuples as a string, or []>",
"score": <1 or 0>
}
A virtual judge that hasn’t been calibrated is worse than no judge at all, because it gives you false confidence. Here are the calibration steps we recommend:
Human judgment remains the gold standard for ground truth, high-stakes domains, and resolving disagreements between automated evaluators.
Overall, the rule of thumb is to start with 20–100 rows labeled by subject-matter experts. Move to a scaled annotation workforce only when the rubric is rock-solid and volume is the bottleneck.
And if your experts disagree on a label, stop. Solve human disagreement before automating anything.
Agentic systems involve multi-step reasoning, tool calling, branching logic and intermediate state transitions. Evaluating only the final output is insufficient: a correct final answer can mask a broken reasoning path, wrong tool parameters, or an inefficient trajectory.
Therefore, you will need to evaluate across three layers:
To achieve this, you can take advantage of the fact that an agent generally pushes traces and spans under an application root. This contains information about the type of agent, the sub agent if invoked, input/output of the agent, tools invoked if any, and more. These traces can be written out to an observability platform or persistent storage.
Then, you can use DFS or another type of tree traversal to reconstruct the trace in memory. This lets you ensure certain subagents were invoked at the right time, the agent called the right tools, etc. And you can scope your evaluation to specific agents/subagents.
Here’s what the full process looks like end-to-end, using a fictionalized and simplified version of a real use case.
Scenario: You’re building an AI assistant that answers questions about a travel platform’s support policies.
Step 1: Explore & discover. Run 100 inputs through your prototype and read every output. You find:15 responses generated policy details not in the source documents (faithfulness issue); 8 correct but too verbose (conciseness); 5 refused valid questions (over-refusal); 3 had broken JSON (format).
Step 2: Build evals. Add programmatic checks for JSON validity and length bounds. Write a virtual judge for faithfulness (separate prompt, different model, chain-of-thought) and another for conciseness. Have your PM or subject matter expert label 60 examples, including failures, as a golden set.
Step 3: Calibrate & iterate. Your faithfulness virtual judge agrees with the PM 78% of the time. Not good enough. Analysis reveals the judge is penalizing accurate paraphrases as “unfaithful.” Update the rubric and add few-shot examples. Agreement jumps to 88%. Improve the retrieval step; faithfulness failures drop significantly.
NOTE: Here, we find that when iterating on models and prompts, it’s best to fix one variable at a time. First fix the model and vary the prompt, then fix the prompt and vary the model, then fix both and vary the serving configuration. At each stage, virtual judge results narrow the candidate pool. Then, you can improve the virtual judge(s) using samples from the top candidates. The evaluators and the candidates sharpen each other until both stabilize.
Step 4: Scale & monitor. Scale evaluation across 5,000 examples. Set up production monitoring: sample 5% of live de-identified traffic daily, run programmatic checks + virtual judges, and surface flagged outputs for human review. A weekly PM review closes the loop, with new failure modes introducing new evals and subsequent system improvements.
NOTE: We sample live traffic continuously using privacy-preserving techniques. All data undergoes robust de-identification prior to human review, and usage is strictly purpose-limited to safety and quality assurance, aligning with Airbnb Privacy Principles.
If this type of work interests you, check out some of our related positions!
We would like to thank Tania Myronivska, Haozhen Ding, and Sebastian Wickenburg for their thoughtful feedback and contributions to this guide, Jisheng Liang and John Hewson for their guidance and insights, and Min Yi and Yi Li for their constant support.
We would also like to thank Evelyn Xu for their support in authoring this post during their time at Airbnb.
All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.
Eval-driven development: Lessons from evaluating GenAI at scale was originally published in The Airbnb Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Continue reading on the original blog to support the author
Read full articleAirbnb demonstrates how to scale Transformer models for recommendation systems by balancing long-term intent with short-term behavior. Their optimization techniques for training efficiency and decoupled inference provide a blueprint for handling massive, sparse event sequences in production.
LLM evaluation is often slow and non-deterministic, making it hard to trust results. By implementing deterministic caching and diagnostic framing of noise, engineers can iterate on models in hours instead of weeks, ensuring that improvements are meaningful and stable for production.
Traditional forecasting fails during unprecedented shocks. This approach demonstrates how to maintain model accuracy in data-scarce environments by using Bayesian prior propagation and cross-geographic signals, providing a blueprint for handling asynchronous global disruptions.
Viaduct offers a middle ground between monolithic GraphQL and complex Federation by allowing teams to contribute to a shared schema via modules. This reduces operational overhead while maintaining developer autonomy, making it easier to scale data access across large organizations.
