This Shit is Hard: Building a vulnerability database from private reports

When a vulnerability is reported in an open source dependency, your team needs to know whether the version you run is affected and what update addresses it. A finding against one release does not answer those questions for everyone using other versions of the package.

Athena brings together vulnerability findings from across the industry so coalition members can coordinate remediation, mitigations, and disclosure. Most of these findings arrive before a public advisory or upstream fix exists. To date, Athena has validated 21,175 zero-day vulnerabilities. Turning those findings into information that guides remediation requires understanding which defect each report describes, where it exists, and what has already been done to address it.

Consider two reports about the same package. One describes a vulnerability in an older release; the other describes similar behavior in the latest version but points to a different function. They could be separate vulnerabilities, or two paths to the same one. An upstream change may already fix one release line while leaving another affected. Those distinctions determine which releases need a patch and whether an existing fix can be used.

In previous posts, we covered how we sandbox Athena's workloads, where Athena’s patches live, and how we generate and test patches when no upstream fix exists. This post covers the vulnerability database that guides Athena. We analyze reports against the source, identify duplicates, determine affected versions, and connect findings to public advisories. Each step helps turn a private report into information a customer can use without having to repeat the investigation.

Understanding the report while protecting the submitter

One of the most important aspects of all this is protecting the sensitive information in a submitter’s environment. A shared vulnerability record needs to explain the vulnerability without exposing the organization that reported it. A report can contain both useful technical evidence and private details about the submitter's environment. We need to separate them before the analysis becomes a shared record.

The analysis pipeline does not receive the customer's identity from the intake service. It receives an opaque submission handle and retrieves the report using a key scoped to that request. The report itself can still contain identifying information, so this separation is only one part of the confidentiality process.

An analysis agent checks out the upstream source at the reported version. It identifies the weakness, the relevant code, the conditions required to reach it, and the path from an entry point to the vulnerable operation. It also searches for other occurrences of the same pattern in the codebase. The agent has read-only access to the source; patch generation occurs in a separate stage so that we can assess the analysis independently of any proposed fix.

Before the description enters the database, a secondary confidentiality check looks for organization names, internal hostnames, ticket references, scanner run identifiers, and other submission-specific details. Information about the public upstream package belongs in the record. Information identifying the submitter or their environment does not.

A separate completeness check requires enough information for downstream processing, including a summary, a confirmed code location, and a Common Vulnerability Scoring System (CVSS) vector. Records missing required information are held for further processing.

Both checks are important. Removing private details must still leave enough technical information to investigate and remediate the vulnerability.

Recognizing the same vulnerability across different reports

Another problem we had to solve was the recognition of repeated vulnerabilities across different reports. Different submitters can describe the same vulnerability using different wording, entry points, or package versions. Recognizing those duplicates lets us connect the reports and their evidence. Getting the match wrong can hide a separate vulnerability that still needs a fix.

Public vulnerability databases also deal with duplicates and overlapping advisories. Athena has the additional requirement of keeping submission details private while resolving them. A customer should not learn who else submitted a finding or how many submissions we received for it.

Our initial approach used a fingerprint built from the package, vulnerable class and method, and weakness type. We also used embeddings to find reports with similar descriptions. Both approaches have limits: two separate vulnerabilities can share a method and weakness type, while two reports about the same defect can identify different functions. Matching fields or wording can suggest a duplicate without establishing one.

We now use an agent to compare an incoming finding with existing records for that package and inspect the source. It evaluates whether the reports describe the same defect and returns a decision with a rationale. The comparison needs to account for the vulnerable behavior and the code responsible for it. Affected-version overlap alone is insufficient: separate vulnerabilities can affect the same versions, and a duplicate report can identify an affected version we have not recorded yet.

Validation code checks the agent's output and saves references before applying the decision. The agent cannot create identifiers or update records directly. These checks catch invalid output and enforce database rules; they do not independently establish that the duplicate decision is correct.

We process submissions for each package sequentially, so two reports cannot both pass the duplicate check before either record exists. If we later merge duplicate records, we retain the withdrawn record and a reference to the retained record, so existing references remain usable.

Determining which versions need remediation

Every organization uses different software versions depending on its unique needs. A finding assessed at one version does not establish the full affected range. The vulnerability may have been introduced recently, existed for years, or already been removed on another release branch. Organizations using different releases need answers that apply to their versions.

The initial record uses a placeholder range covering versions up to and including the reported version. That range is provisional. It can include older versions that are not affected, and it does not establish whether later versions are vulnerable. Further analysis is needed before it can accurately direct remediation.

An enrichment agent examines the vulnerable behavior and traces it through the project's Git history. It maps release lines, looks for the change that introduced the behavior, and searches for changes that removed it. Commit messages are not enough, as an ordinary refactor can remove a vulnerability without mentioning security.

And release branches make this even more complicated. A fix in the 2.x release line does not establish whether 1.x is fixed. A backport may implement the same correction in a different commit, and a change on the main branch may not yet appear in any published release. The analysis needs to connect the relevant source changes to the package versions customers can actually use.

Finding a related change also does not establish that it is a complete fix. It might address one entry point while leaving another exposed, or require additional changes to remove the vulnerable behavior. Choosing the wrong boundary could lead a customer to believe an upgrade addresses the vulnerability when it does not.

A second agent reviews proposed fix boundaries. It checks whether the change removes the vulnerable behavior and whether the identified releases contain that change. Additional validation checks use package-registry versions and repository history to verify version-to-reference mappings and commit relationships. The analysis runs in an isolated microVM without credentials or outbound network access while its tools process repository content.

These checks answer different questions. Repository validation can establish that a release contains a commit. Source analysis must still establish that the change addresses the reported defect. A second agent's agreement does not, by itself, prove that the conclusion is correct.

Range updates must also preserve evidence already recorded. An update must not reclassify a confirmed fixed version as affected or remove a version confirmed vulnerable by a submission. Conflicting conclusions need to be resolved before the range changes.

Matching records to public advisories

A private finding may later receive a public identifier (e.g., CVE). It may also describe a vulnerability that was already public by the time the report arrived. Missing the public identifier in the submission does not establish that the vulnerability is new.

Matching these records requires comparing the package, weakness, vulnerable code, and attack conditions. When a public advisory describes the same vulnerability in the same package, we can add its identifier as an alias to the existing record. This connects the private finding to information customers may encounter through public advisories and their security tools.

A public advisory and an upstream fix are separate pieces of information. An advisory may describe an unresolved vulnerability, and a fix may exist before an advisory is published. Adding an alias does not establish that a fix is available. The remediation pipeline still needs to identify and validate the upstream change before using it.

The work behind a remediation decision

Customers may see an affected version range and a fixed version. Behind those fields are decisions about whether two reports describe the same defect, whether a change actually removes it, and which releases contain that change. Those decisions have consequences: a separate vulnerability can disappear into an incorrect duplicate match, an affected release can be left out, or a version that still contains the defect can be marked fixed.

That is why this database is part of delivering remediation. We need to turn incomplete reports and changing repository history into records that guide patching, preserve private information, and make unresolved questions explicit. AI agents help investigate the code, validation checks control record updates, and evaluation shows where the resulting answers need improvement.

The work is difficult because each conclusion must hold for the package version a customer actually uses. Chainguard takes on that investigation so customers can decide what to update without having to reconstruct the vulnerability's history themselves.

Learn more about Athena and how we can help you protect your organization from AI-assisted attacks.