Metrics Board solves the 'data trust' problem by unifying metric definitions and automation. It transitions metrics from ad-hoc scripts to production-grade code, ensuring consistency across experiments and providing the structured metadata necessary for reliable AI-driven analytics.
Michele Ceccacci; Software Engineer I | Jason Coffman; Sr. Software Engineer | Colm O’Shaughnessy; Software Engineer II | Laura Palmer; Staff Product Manager | Adam Podraza; Manager, Engineering | Surya Karri; Manager, Engineering
At Pinterest, reliable and trustworthy metrics are behind every decision: from measuring company-wide business performance to evaluating each feature experiment results. Hundreds of data producers — analysts, data scientists, and engineers across Pinterest — create thousands of metrics from our petabyte scale data lake. A metrics ecosystem this size can’t be ad hoc; it has to be organized, scalable, and resilient. So we built Metrics Board: an end-to-end platform that automates metric creation, enforces quality and enables agent-driven analytics.
In this post we walk through the product experience, architecture, integrations and explain some of the design choices we made to make metrics work at planet scale.
Most organizations rely on a familiar set of metrics — revenue, growth, engagement — to steer the business.
Metrics are how we understand whether Pinterest is delivering value to users and advertisers. Measures related to areas such as revenue, growth, and engagement turn the complexity of user behavior and Pinterest performance into a shared view of what is working, where we are falling short, and what matters most. These metrics drive the decisions our product, finance, and executive leaders make every day, from what to build to how we forecast and steer company strategy.
Metrics also power Pinterest’s product experimentation practice, which runs at massive scale — thousands of experiments are active at any given time. Those experiments are used as gates for each new feature launch. For results to be fair, comparable, and statistically significant, metrics must be clearly defined, reliable, and reusable. Beyond common core metrics, individual experiments often need bespoke metrics scoped to a single module, feature, or device. For example, a feature to add a new form of interaction with a Pin might need its own measure to evaluate engagement with the feature — even if core product engagement metrics are flat, the experiment might be successful if it creates a new form of user-engagement that a new metric would need to measure.
As much as key decision-makers need metrics, the way they get answers from data has changed too. Increasingly, the question isn’t typed as a SQL query or a dashboard filter — it’s asked in natural language to an agent. But agents are only as good as the trusted data they can access. For agentic analytics to work, metrics have to be discoverable, quality-checked and machine-readable.
In the past, users were defining their metrics in bespoke data pipelines and wiring them by hand into our reporting and experimentation products. That approach had several problems:
Put together, these problems slowed teams down, held experiments back, and left us with more metrics than we could trust. We needed a solution that would:
Our goal was to make metrics easy for everyone who touches them — the people who produce them, the people who consume them, the applications that serve them, and, increasingly, the agents that reason over them.
Define a metric once, Metrics Board handles the rest.
We built Metrics Board as a thin semantic layer for metrics on top of the data tools Pinterest already runs: a metric is defined once, and that definition drives everything downstream. It’s built around five pillars: inventory, compute, quality, publishing, and serving & discovery.
The inventory pillar is our single source of truth for metric definitions. Every metric lives as a YAML spec in a dedicated Git repo, and is treated like any other production code, with version control, code review, and change history built in.
version: 1
metric_id: mb__metrics
display_name: Metrics Board metrics count
description: |-
Count of metrics defined in MB. Test metrics aren't included.
domain: tech_data
metric_sql: |-
with mb as (
SELECT urn
FROM query_analytics.pincat_db_latest_state_v2
WHERE dt = '{end_date}'
AND aspect = 'structuredProperties'
AND is_active
)
SELECT
COUNT(mb.urn) as mb_metrics,
'{end_date}' as dt,
FROM mb
temporal_columns:
- column_name: dt
format: YYYY-MM-DD
granularity: DAY
integrations:
superset:
enabled: true
starrocks:
enabled: false
ownership:
nimbus_project: ii
compute:
schedule: 0 9 * * *
One important consideration at the time of design was how much should Metrics Board be a true semantic layer. Do we enforce a model on top of existing datasets and have metrics defined as expressions leveraging the model? Or do we accept any SQL query and ask users to provide modelling information (temporal columns, metric or dimensions definitions) for the resulting data? We decided on the latter, driven by two main considerations: user feedback (“Put Pinners First” is one of our company values) and the sheer variety of tables which are used for metrics definitions — enforcing a semantic layer for each was simply not realistic.
We don’t ask teams to craft definitions by hand. The Metrics Board UI is a dedicated interface: users define metrics through guided forms, and the system generates the corresponding YAML and opens a pull request on their behalf. The result is one place to manage ownership, governance and metadata, that still integrates cleanly with the rest of the data stack.
To compute metrics, Metrics Board turns each definition into its own scheduled Airflow DAG. A definition carries all the metadata we need — schedule, SQL, validations, integrations — to generate a dedicated workflow. Isolating each metric in its own DAG lets us tune schedules, retries, and resources individually, and prevents a failure in one cascading to others.
At execution time, Metrics Board compiles each metric’s SQL into SparkSQL or PrestoSQL depending on the engine named in the definition. It then analyzes the SQL code to automatically create dependency tasks which wait on upstream data to be ready (we rely on sqlglot for parsing). It also automatically generates standard data quality checks and lets users define additional ones too (more on that below). Backfills are managed through Airflow, so owners can recompute history easily. Metrics Board also provides a layer for automated operations. For example if upstream data is late, Metrics Board automatically detects when it lands and resumes the metric computation DAG.
Taken together, the above listed features make metrics creation and management easy even for non-data teams. What used to take days or weeks to build can now be achieved in 1–2 hours — a significant boost for users’ development and experiment velocity.
A metric nobody trusts is worse than no metric. Every definition can declare validations that run as part of the metric’s own pipeline, backed by two engines:
When a check fails, the right people hear about it in real time. Alerts fire as soon as validation runs, route to the owning team, and can be snoozed or acknowledged so the signal stays meaningful and doesn’t turn into noise.
All of this rolls up into a quality report per metric — per-check pass rates, a time series of results, gaps where validation didn’t run, and annotations (comments or linked tickets) — available via API and as a PDF export. Because the validations live in the same spec as the metric, quality is versioned and reviewed alongside the definition itself.
A metric that lives only in its own pipeline isn’t useful yet — it has to reach the products people actually work in. Publishing does that automatically, driven entirely by the definition. Each metric declares its integrations, and turning a surface on is as simple as a flag: enable it, and Metrics Board fans the metric out to that surface with no per-surface wiring by hand. Today a single definition can publish to:
Publishing happens in two phases:
Either way, the outputs (dataset IDs, the dashboard link, the output table, the pipeline URL) are written back to DataHub, so the catalog stays the single place to find a metric and everything attached to it.
Discovery & governance. Metrics Board is where the org finds and evaluates metrics — but consumers have to be able to find a metric before they can evaluate it. Metrics Board centralizes definitions into a single source of truth, capturing rich metadata alongside them: ownership, description, certification grade, and quality signals.
That metadata is what turns centralization into discovery. Every metric automatically flows into PinCat, so users can discover them in the same place they already go for datasets. With metrics easy to find, the next question is what to trust. Two signals do that work, and they operate at two levels.
At the metric level, every metric carries a grade — a certification level that climbs from Unknown through C, B, and A up to A+ (fully certified and manually verified by a Data Quality Steward), with each rung demanding more quality work. It tells consumers what to trust and shows owners exactly how to level up. Furthermore, a metric’s key metadata — grade, certification state, and owner — is surfaced directly within its Superset chart.
At the reporting level, dashboards in Superset are classified into their own three tiers that reflect how critical and durable they are, and a dashboard’s tier drives three concrete things: how long it’s retained when unused, which Presto cluster its queries run on (and therefore its performance), and what documentation it must carry.
Together, metric grade and dashboard tier give consumers a trust signal at every layer of the stack, while retention and query-lane routing fall out of the dashboard tier automatically.
Agent enablement. The same structured metadata that powers discovery makes Metrics Board a foundational platform for analytics agents. We expose an MCP server so any LLM-based tools can find metrics, read their definitions, and reason about them. The clearest example is our Analytics Agent, which can answer natural-language questions about metrics. By referencing golden queries and certified metric definitions, the agent ensures its responses are grounded in trusted, governed data. As it turns out, the work we did to make metrics trustworthy for humans is what makes them usable by agents.
Adoption is the primary success measure we track, and we’ve exceeded adoption goals quarter after quarter — both in metrics created and in active users. Today nearly 150 new metrics are created each month (we also built out a lifecycle policy and engine to garbage-collect metrics no longer in use). Over 98% of metrics created for experimentation go through the Metrics Board today.
That adoption rides on dramatically improved velocity. On one recent high-profile project revamping key engagement metrics, Metrics Board saved roughly two engineering-months versus the old approach; metric additions that once took weeks now take days. At scale those savings translate to thousands of hours saved each month.
And velocity isn’t the whole story: the governance, quality, and cataloging we built along the way are quickly becoming the foundation for the LLM and agentic use cases we’re now shipping.
Analytics Platform: Krishna Gopal
Workflow Platform: Dinghang Yu, Martin Yau
Metrics Quality: Dennis Marcu, Raj Phull
Experiment Platform: Lu Yang, Xinyue Cao
Real-time Analytics: Charles Wu
Metrics Board: Building an Agent-ready Metrics Layer was originally published in Pinterest Engineering Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Continue reading on the original blog to support the author
Read full articleThis approach shows how to collapse complex multi-stage recommendation pipelines into a single generative task. It proves that distilled 4B-parameter LLMs can provide production-grade performance for high-scale personalization, significantly improving user intent understanding and engagement.
Determining when a streaming dataset is ready for batch processing is difficult. This metadata-driven approach signals data completeness based on event-time watermarks, reducing processing latency while ensuring data integrity against late-arriving records.
Standard retrieval models often collapse diverse user interests into a single vector. This architecture demonstrates how to scale multi-intent retrieval using transformers and GPU-optimized batching, significantly improving relevance and engagement in massive-scale recommendation systems.
Scaling recommendation models is often limited by network bandwidth rather than compute. This demonstrates how to overcome communication bottlenecks in embedding-heavy architectures, enabling massive model training with near-linear efficiency and optimized infrastructure costs.
