This article provides a blueprint for scaling specialized institutional knowledge using AI. By separating reasoning from knowledge and enabling feedback loops without retraining, organizations can build reliable, auditable, and self-improving expert systems that preserve human expertise.
Many large organizations have the same problem when it comes to specialist knowledge. While some of it is written down in the form of models, playbooks, checklists, and frameworks, the most valuable specialist knowledge lives in people’s heads and rarely gets captured anywhere durable. In compliance domains, for example, the same types of questions can arise across hundreds of product reviews, expert assessments take days of manual research, and inconsistency between assessments creates real organizational risk.
It’s not uncommon for experts to spend more time answering routine questions than on genuinely novel and ambiguous work where their judgment matters most. We need systems that can capture how an organization’s experts reason and make that knowledge available to everyone who needs it, so that expertise is easier to share, build on, and preserve.
We set about solving this challenge by codifying institutional intelligence into an AI agent for a specific compliance domain. The agent combines a knowledge system that acts as the organization’s “second brain,” a reasoning layer that mirrors how domain experts actually think, and an automated improvement pipeline that compounds expert effort permanently. The patterns generalize to any enterprise domain with deep specialist knowledge, whether that is finance, security, or engineering.
Off-the-shelf LLMs provide a strong foundation, but they often need deeper institutional context to be fully effective in specialist domains. Without that grounding, a general purpose model has limited value given it will not be able to distinguish between what an organization could do (a summary of general information) and what it should consider doing (based on historic positions, company direction, business context, etc.). In high-stakes domains, closing this gap requires supplying the model with the organization’s own knowledge and priorities so its analysis reflects how the organization actually reasons.
The system we’ve designed has four layers, each solving a distinct problem:
These layers depend on each other. The knowledge system’s file structure makes automated editing possible. The reasoning layer’s explicit procedures make failure attribution tractable. The evaluation framework gates every change. And the improvement loop feeds back into both knowledge and reasoning. Remove any one layer and the others degrade.
Large organizations can accumulate thousands of documents as a byproduct of expert work. It is tempting to treat those documents as organizational knowledge, but the real knowledge is implicit: how experts reason, what they prioritize, and how they resolve ambiguity. An agent that retrieves document chunks at inference time has to re-derive that reasoning from raw sources on every run, which is slow, error-prone, and inconsistent.
We make that implicit knowledge explicit ahead of time. A long-running offline process reasons through source documents and distills them into structured knowledge files – curated statements of how the organization interprets its domain, with constraints, boundaries, and routing implications made machine-readable.
Most significantly, that knowledge then forms the basis of a feedback loop that allows the agent to learn from and implement feedback from human experts without the underlying model having to be retrained.
The industry has converged on a similar idea. Andrej Karpathy’s LLM Wiki structures agent knowledge as a navigable graph of files, and Google’s Open Knowledge Format standardizes this for cross-agent interoperability. The shared insight is that knowledge should be pre-extracted, explicitly structured, and progressively disclosed rather than re-derived on every query. We extended these principles into a system where citation fidelity and institutional consistency are non-negotiable, organizing 200+ files into a strict taxonomy:
Every file declares its dependencies (depends_on) and consumers (referenced_by) in YAML frontmatter, forming a bidirectional dependency graph. When one file changes, you can trace exactly what else might be affected, which matters when the self-improvement loop proposes automated edits.
A key architectural decision is how to partition knowledge between the curated wiki and supplementary retrieval (RAG). We split on information density and expected usage frequency.
High-density, frequently referenced sources go into the wiki: Distilled files capturing how the organization reasons, such as positions, decision frameworks, boundary examples, and strategic interpretations. The agent consults these on nearly every turn. Because they encode the organization’s evolving thinking, they need to stay current, and the wiki structure makes them easy to update, version, and validate.
Sparse, situationally relevant sources are served through semantic or lexical search (RAG): documents that matter deeply when they apply but are not needed in detail on most runs, such as detailed reference material, individual product specifications, historical decision records, and niche external knowledge. Loading all of them into the wiki would bloat the system and dilute attention.
The result is that the agent’s core reasoning is always grounded in the most refined, current organizational knowledge, while it can still reach for supporting evidence when a scenario demands it. The combination produces an organizational second brain that encodes how the organization interprets and applies information, rather than only where to find it.
Knowledge alone is not enough. Domain experts do not simply recall facts, they follow structured methodologies: a financial analyst works through a valuation model step by step, a security engineer follows a threat modeling procedure. The challenge is capturing those methodologies in a form an LLM can execute reliably.
We solve this with composable procedures we call recipes. Where knowledge files are declarative, recipes are imperative. Each one prescribes a multi-step analytical workflow, specifying what to examine first, which knowledge to load at each step, what decision procedures to follow, and what constitutes a complete analysis.
The critical design choice is separating what the agent knows from how it reasons. Recipes reference knowledge files but contain no domain facts; knowledge files state positions but prescribe no procedures. This means:
Recipes compose into pipelines, much like a head chef’s master recipe for a dinner service delegates to sub-recipes for each component (the sauce, the protein, the garnish) without containing those details itself. Our top-level routing recipe examines the input and selects which downstream recipes to invoke, each handling one analytical phase.
This is also what enables progressive disclosure. Rather than front-loading a monolithic instruction set covering every possible scenario, each recipe step carries only the instructions and knowledge relevant to that phase. Early versions used a single flat instruction file and loaded all sources via semantic search, pulling a large volume of mixed-relevance files into the context window on every run. After restructuring into recipe-driven stages, each query touches only a small, targeted subset, cutting tokens consumed per turn by around 80%. Context windows are finite and attention degrades with volume, so delivering the right instructions at the right time directly improves reasoning quality.
Human experts stay in control of this system throughout. The agent accelerates and structures their work; it does not replace their judgment or their authority over the outcome.
We enforce this through two mechanisms:
Checkpoints are defined points in the analysis where the agent surfaces its intermediate reasoning for expert review before proceeding, and the expert can confirm, correct, or redirect.
Escalations trigger when the agent hits genuine ambiguity, whether from underspecified inputs or evidence that supports more than one defensible reading. Rather than forcing a resolution, it hands the question to the expert, whose choice determines the path the analysis takes.
Checkpoints and escalations serve three purposes simultaneously:
The more consequential the decision, the more this matters, which is why we recommend keeping a human in the loop by default across domains like compliance, financial risk assessment, security review, and engineering safety.
We consider the self-improvement flywheel to be the most distinctive part of this system. While the structured knowledge system and composable recipes produce a system that is legible by humans and agents, testable, and modular, the number of interdependent files make manual maintenance impossible to scale. When domain experts provide feedback to an agent that feedback has to be translated into precise file edits. This process can take weeks because it requires understanding the full dependency graph, verifying nothing else breaks, and validating the fix actually works.
There has been a large body of work – from RAG memory systems to model-weight knowledge editing – devoted to addressing how agents store, retrieve, and update knowledge. But much less attention has gone toward keeping a document-based institutional knowledge base correct as it grows and as expert positions evolve. Agents that auto-draft their own fixes are increasingly common; but we haven’t seen this level of validation rigor applied to a structured knowledge base without model retraining.
We treat that maintenance as a compilation problem and automate it. Every expert correction moves through four phases:
Once the loop completes, the regression test suite is enriched with the issue that was just fixed so that future updates preserve this behavior.
Raw expert feedback comes from conversation traces where domain SMEs interacted with the agent and provided corrections. The diagnosis phase extracts structured signals from these conversations.
Our first approach classified feedback by conversational form. If the expert provided information, it must be a knowledge gap; if they redirected the agent, it must be a procedure problem. This heuristic failed because conversational form is a poor proxy for root cause. An expert correcting a conclusion might be exposing a knowledge gap, a recipe flaw, or a genuine ambiguity.
The working approach separates extraction from classification. First, extract every substantive signal from the expert alongside the agent’s full knowledge manifest (every file loaded, when, and how used). Second, read the actual knowledge files and apply a single attribution test: Could the agent have reached the correct conclusion from its source materials?
The compiler translates each diagnosed issue into minimal file edits. Sub-agents analyze impact in parallel, examining cross-references, conflicts with existing positions, token budget impact, test coverage, and duplication risk.
Two design choices make this trustworthy:
Independent adversarial review. A separate agent, running in a fresh context with no knowledge of the improvement rationale, receives only the proposed diffs to the knowledge base. Its job is to find problems such as contradictions introduced, edge cases broken, or positions undermined. Because it shares no context with the proposing agents, it cannot inherit their blind spots.
Deterministic structural validation. A linter catches issues programmatically, dangling cross-references, file size budget violations, identifier collisions, and dependency cycles. This layer is not probabilistic. It passes or fails.
Every proposed change goes through a two-stage validation:
Targeted replay runs the agent on the original scenario that triggered the feedback. The agent does not know it is being tested. A separate judge evaluates the new output against the original expert feedback without knowing what was changed. This deliberately blind design prevents confirmation bias. If targeted replay fails, compilation is retried.
Regression testing runs multiple benchmarks for that domain, which are usually structured test suites of Q&A pairs. For analytical domains whether there might be multiple correct answers, an independent LLM judge gives each test case a pass/fail based on certain criteria. The agent is run in parallel, independent sessions against the benchmark questions, and regressions in performance are detected. If regression testing fails, compilation is retried with an updated prompt describing where the agent regressed, along with the original issue and attempted fix.
The output of the pipeline is a pull request (diff) with a complete audit trail. A human expert reviews a proven fix rather than debugging a raw failure. Once approved and landed (the knowledge file or recipe is updated), the original failing scenario and its validated correct answer are automatically added to the regression test suite. This means every fix permanently raises the bar and future changes to the knowledge system must preserve the behavior that was just corrected.
After three development sprints spanning six weeks, the system achieved:
The specific domain we built this for required synthesizing dozens of sources (both internal positions and external material) into risk-weighted assessments. But the architecture is domain-independent. It applies wherever:
Concrete domains where this pattern fits include regulatory compliance, protocol adherence, financial risk assessment, security review, engineering standards compliance, and procurement evaluation. The common thread is that organizations need AI systems with genuine institutional expertise, not just general knowledge.
The requirements for adopting this architecture are:
The deeper principle is straightforward: keep the complexity in text files that are readable by both humans and agents, rather than fine-tuned model weights. Every improvement is a text edit that a domain expert can review in 30 seconds. Every change is version-controlled, diffable, and reversible. The compilation pipeline is sophisticated, but its outputs are always transparent.
The goal is a system where expert effort compounds permanently. Every expert interaction makes the system better. Every correction persists as a verified improvement. The organization’s collective knowledge stops being trapped in individuals and starts being available, consistently and at scale, to everyone who needs it.
The authors would like to express our gratitude to the contributions of the following people, who have played a crucial role in developing this system. In particular, we would like to extend special thanks to (in last name alphabetical order): Cecilia Baek, Philipp Kaufold, Cat Hughes, Suzanne Leijten, Michael Marcusa, Jordi Mola, Timothy Neo, Elliott Prentiss, Laia Reyes, John Ross, Julio Santil, Taylor Wilson Thomas, Mansi Tripathi, Nikhil Shanbhag, Madeleine Vos, and Jackie Zajac.
The post An Organizational Second Brain: Building an AI That Learns From Experts appeared first on Engineering at Meta.
Continue reading on the original blog to support the author
Read full articleThis architecture solves the trade-off between model complexity and serving latency. By decoupling user modeling from ranking, engineers can scale transformer capacity and sequence lengths predictably, achieving LLM-like performance gains in high-throughput production environments.
Meta's GEM shows how to scale recommendation systems to LLM levels. It proves that standard LLM optimizations fail for recsys due to jagged sequences, requiring hardware/software co-design and custom kernels to achieve high GPU utilization and 4x FLOPs scaling.
This research addresses the challenge of sparse signal optimization in massive-scale recommendation systems. By using hierarchical graph learning and multimodal enrichment, engineers can improve deep funnel performance and better align user intent with content in high-sparsity environments.
Storage bottlenecks are a primary cause of GPU stalls in AI workloads. Optimizing BLOB storage for low-latency retrieval is critical for maximizing expensive compute utilization and accelerating the development of frontier models.
