Secret leakage is a major security risk, but high false-positive rates lead to alert fatigue. GitHub's ML approach demonstrates how to use context-aware filtering to make security tools more accurate and less intrusive, improving the overall security posture without slowing down development.
OUR PIPELINE VS. THE PREVIOUS LLM PIPELINE
Secret scanning works best when developers are confident in the findings. Catch too little, and real credentials can remain in a repository. Alert too often, and developers may spend time investigating strings that aren’t secrets.
Generic secrets make that balance especially difficult. The same string could be a real credential or a documentation placeholder, so detecting it requires understanding the surrounding code and text.
To address this challenge, we trained a secret classifier specifically for GitHub’s product needs: distinguishing real credentials from lookalikes while meeting the speed and efficiency requirements of scanning at scale.
In our companion post, Secret protection must scale with software, we explained why secret protection must keep pace with AI-accelerated development. Here, we go behind the scenes of the context-aware classifier powering that effort: how we designed, trained, and evaluated it to find more real secrets with less noise.
GitHub secret scanning operates at two key moments:
We developed a fast classifier that reduced median post-push scan execution time by 86% compared with the previous LLM-based pipeline. That efficiency made it practical to extend generic secret detection to push protection, returning results before developers push changes, not only after code is committed.
Structured credentials often include a distinctive prefix, length, alphabet, or checksum. Generic secrets may not. Their syntax overlaps with build identifiers, checksums, asset hashes, sample values, and other non-sensitive strings. Context is the deciding signal.
| Likely real credential | Likely harmless lookalike |
|---|---|
| A password assigned beside a production database URL | The same password-shaped value in a test fixture |
| A connection string in deployment configuration | A localhost connection string in a setup guide |
| A high-entropy value bound to an API credential | A high-entropy checksum, build ID, or asset hash |
These cases are hard negatives: strings that match a credential pattern but are not usable credentials. Adding more rules can suppress some of them, but each exception risks hiding a real secret. A learned classifier can instead use signals such as file path, secret type, variable names, and nearby code or text.
That is why we shipped a new scanner using a two-stage pipeline:
Impact:
As content is scanned, regex patterns cast a wide net and flag anything that looks like a credential, so real secrets aren’t missed at the start. Each candidate then goes to a fine-tuned ModernBERT classifier, which reads the value together with its file path, variable names, and surrounding code. The classifier estimates how likely it is to be a real secret. High-confidence findings become alerts, and lookalikes such as placeholders, test fixtures, and checksums are filtered out before they reach developers.
HOW OUR PIPELINE WORKS
The important design choice is that the model does not replace deterministic scanning. Instead, it complements it. Regexes remain the right tool for finding candidate strings quickly. The classifier is used where machine learning provides the most leverage: distinguishing “looks like a secret” from “is likely a real secret.”
Most false positives are not random. They are hard negatives: strings that match a credential pattern but are not usable credentials. Examples include documentation placeholders, toy examples in tests, redacted values, non-sensitive IDs, and generated fixtures. These cases are easy for a regex to match and hard for a purely rule-based system to suppress without accidentally losing real secrets.
ModernBERT helps because it can read the candidate together with nearby code and text providing contextual understanding comparable to an LLM without the latency and computational overhead of generating natural-language output. For example, a candidate surrounded by words like “example,” “dummy,” “fake,” or “replace-me” may be lower risk than a candidate assigned to an environment variable in a deployment file. The classifier learns these contextual differences from labeled examples rather than requiring engineers to encode every exception as another rule.
We chose ModernBERT because it provides a strong fit for this workload: it is compact, efficient, and well suited for classification over code-adjacent text. For secret scanning, the model needs to make a calibrated decision over short candidate windows with high throughput and low latency.
That makes a fine-tuned encoder classifier a better serving shape than a general-purpose LLM for this stage of the pipeline. The model can be trained directly on our decision boundary, and evaluated using the same recall and precision metrics that matter to secret scanning quality.
The challenge was finding representative data for cases in which syntax alone is misleading. The training and evaluation strategy combines three sources:
Synthetic rows pass deterministic validation and model-based quality review before inclusion. Training emphasizes real secrets, realistic false positives, and hard negatives that frequently confuse regex-only approaches. Each model input includes the candidate value, file path, secret type, and a bounded amount of surrounding context. We select thresholds through offline evaluation and then validate behavior on production-like traffic before broader rollout.
Before splitting the data, we grouped rows that share a repository, candidate value, or example so near-duplicates could not appear in both training and evaluation.
Training focused on the cases that matter most for alert quality: real secrets, realistic false positives, and hard negatives that regexes frequently match. The input representation includes the candidate secret string, file path, type and a bounded amount of surrounding context. The output is a classification score used by the scanner to decide whether the candidate should proceed.
We optimized for the product tradeoff: preserve or improve recall while reducing false positives and serving cost. In practice, that means the classifier threshold is selected using offline evaluation, then validated against production-like traffic before rollout.
The most important training principle was to make the model see the confusing examples. We included candidates that are syntactically similar to real secrets but semantically different, so the model learns not only what a secret looks like, but also what a false positive looks like in context.
The trained model is exported to ONNX format with 8-bit quantization, enabling fast inference.
| Online result | Change vs. previous LLM pipeline | What it measures |
|---|---|---|
| Repository owner-weighted acceptance | 11.5% higher | Customer acceptance of detected findings |
| Median post-push scan execution time | 86% lower | Production scan execution time |
| Serving cost | More than 98% lower | Cost of the compact classifier compared with remote LLM calls |
We evaluated the new pipeline’s impact in production in two stages: first against the production baseline on the same public traffic and then through a customer A/B experiment.
The best security tool is the one developers trust enough to use. Higher precision means fewer distracting alerts. Higher recall means more real secrets are caught. Lower cost means we can keep improving coverage without making scanning prohibitively expensive.
For developers and security teams, this should translate into fewer false positives, and faster feedback in the places where secrets are most likely to be introduced: commits, pull requests, and everyday development workflows.
Hybrid systems are powerful. Regex gives us speed, transparency, and broad candidate coverage. ModernBERT gives us learned context understanding and false-positive reduction. The combination outperforms either approach used alone for this product constraint.
We also learned that model size is not the same as model usefulness. A smaller fine-tuned classifier can be the right answer when the task is narrow, the decision boundary is measurable, and serving efficiency is critical.
This is one step toward making secret scanning more accurate, scalable, and easy to use. Looking ahead, we are focused on two areas:
Huge thanks to the engineering, science and product teams across GitHub and Microsoft who helped build the training data, evaluation suite, serving stack, and rollout plan for this new secret scanner.
The post GitHub’s model for secret detection: Finding more secrets with less noise appeared first on The GitHub Blog.
Continue reading on the original blog to support the author
Read full articleAs AI agents accelerate code production, manual security reviews are no longer sufficient. Engineers must adopt automated, low-latency security tools that can detect secrets at the speed of agentic development to prevent large-scale credential exposure.
As AI agents move from experimental to production, engineers must solve critical challenges in context management, deterministic verification, and security. This overview provides a roadmap for building reliable, agentic systems while maintaining robust supply chain integrity.
Policy decisions directly impact the legal stability of open source licenses and the classification of developer infrastructure. Understanding these shifts helps engineers navigate compliance requirements for AI transparency and age-appropriate design without compromising technical autonomy.
This tool demonstrates how AI agents can automate complex security audits by breaking them into logical steps. It empowers engineers to scale vulnerability research across mobile ecosystems, finding critical flaws that traditional static analysis or generic LLM prompts might miss.
