This library provides a production-ready implementation of k-anonymity, allowing engineers to perform statistical analysis on sensitive datasets while mathematically guaranteeing user privacy. It bridges the gap between theoretical privacy models and scalable, high-quality data engineering.
By: Adam Bloomston
In 2020, we launched Project Lighthouse, which we developed in partnership with leading civil rights and privacy organizations. As our 2020 announcement details, Project Lighthouse enables us to measure potential disparities in user experiences. This work uses perceived race data that is never linked to individual accounts; we only use this data for measuring potential disparities, and users who want to opt-out can do so by turning off the data use settings in their account’s Privacy page. Our results, shared in 2024, demonstrate how we use these analyses to measure our progress in mitigating those disparities.
Earlier this year, we open-sourced project-lighthouse-anonymize, the Python library that powers Project Lighthouse’s anonymization process. To provide the full technical foundation for this work, we also published two new papers on arXiv alongside the code release. Together with our original 2020 paper, these three papers form a complete story: the foundational methodology, the scalable implementation, and the quality validation framework.
Our original 2020 paper established the privacy-by-design approach for Project Lighthouse and provides the rationale for choosing k-anonymity as the technical privacy model to prevent sensitive attribute disclosure at scale. For an introduction to this paper, see our first blog post on p-sensitive k-anonymity and our second blog post on measurement with anonymized data.
The first of our new papers, Core Mondrian: Basic Mondrian beyond k-anonymity, presents the k-anonymity algorithm at the heart of the open source library. Core Mondrian extends the classic Mondrian algorithm with:
This algorithm enables scalable anonymization of large datasets while preserving the ability to use the underlying data for statistical analyses.
The second of the new papers, Measuring Data Quality for Project Lighthouse, addresses a critical question: how do you know if your anonymized data are “good enough” for your analysis?
The paper introduces a comprehensive framework for measuring data quality under anonymization, including:
Three primary metrics:
Empirical validation methodology: We reframe data quality assessment as a machine learning classification problem, using synthetic datasets to validate that our metrics and thresholds successfully predict when anonymized data will produce statistically valid results
Default thresholds: The paper and library include the specific threshold values we use for Project Lighthouse, which may serve as useful starting points for others implementing similar systems
This framework enables analysts without deep anonymization expertise to confidently assess whether their anonymized data supports valid statistical conclusions.
The library is available on PyPI and GitHub at github.com/airbnb/project-lighthouse-anonymize. With this library, you can successively enforce both technical privacy models from our 2020 paper:
p, k = 2, 5
# First technical privacy model: k-anonymity
anon_df, dq_metrics, disclosure_metrics = k_anonymize(logger, input_df, qids, k, {}, "row_id")
# Second technical privacy model: p-sensitive k-anonymity via perturbation
sensitized_df, _, _ = p_sensitize(logger, anon_df, qids, "race", p, k, sens_attr_value_to_prob)
The algorithm for enforcing k-anonymity is described in Core Mondrian: Basic Mondrian beyond k-anonymity. And the data quality metrics and thresholds for k-anonymity are described in Measuring Data Quality for Project Lighthouse:
minimum_dq_met, minimum_dq_met_reasons = check_dq_meets_minimum_thresholds(dq_metrics)
assert minimum_dq_met, str(minimum_dq_met_reasons)
The getting started guide builds on the code snippets above and provides a complete, runnable example using the UCI Adult dataset.
We continue to invest in trying to combat potential discrimination and bias users may face when using Airbnb, and in taking steps to enable everyone in our global community to use and enjoy Airbnb. We believe that doing so requires transparency in our methodologies, both to build trust with our users and to encourage other companies to do the same.
If this type of work interests you, check out some of our related positions!
The Airbnb Anti-discrimination & Equity team is Adam Bloomston, Elizabeth Burke, Megan Cacace, Anne Diaz, Wren Dougherty, Matthew Gonzalez, Remington A. Gregg, Yeliz Güngör, Eeway Hsu, Heesoo Kim, Sara Kwasnick, Joanne Lacsina, Demma Rosa Rodriguez, Adam Schiller, Jessica Simon, Maggie Tang, Skyler Wharton, Marilyn Wilcken. I also want to thank Natalija Fijacko, Lauren Mackevich, Laura Rillos, Jessica Simon, Floyd Smith, and Lei Wei for their role in refining and improving this blog post.
The author acknowledges the use of Large Language Models (LLMs) for assistance with literature review, technical writing, and editing.
All product names, logos, and brands are property of their respective owners. All company, product, and service names used in this website are for identification purposes only. Use of these names, logos, and brands does not imply endorsement.
Project Lighthouse — Part 3: Introducing project-lighthouse-anonymize was originally published in The Airbnb Tech Blog on Medium, where people are continuing the conversation by highlighting and responding to this story.
Continue reading on the original blog to support the author
Read full articleScaling graph databases for real-time applications is difficult. Airbnb's move to an internal JanusGraph platform demonstrates how to decouple storage from logic to achieve high performance, reliability, and operational control for massive identity resolution workloads.
This architecture demonstrates how to build social features without compromising privacy. By decoupling internal identities from public profiles, engineers can provide granular user control and prevent unintended data leakage across different product contexts.
Forecasting models drive critical business decisions. Understanding when to update a model versus when to wait prevents costly biases. Airbnb's framework helps engineers distinguish between temporary noise and structural shifts, ensuring long-term model reliability in volatile markets.
Airbnb's server-driven approach allows for rapid experimentation and regional customization without client-side updates. This architecture improves conversion by dynamically selecting the easiest verification method for each user context, significantly reducing friction for infrequent users.
