This Shit is Hard: Building forkstore

The This Shit is Hard series has spent a lot of time on how we make patches: backporting disclosed CVE fixes in Chainguard Libraries, authoring fixes for zero-days that have no upstream patch at all, and running all of it inside a sandbox. This post is about a problem that sits underneath all of it: once you've made those patches, where do they live?

Chainguard Libraries rebuilds open source packages with fixes for publicly disclosed CVEs, ordinary backports of vulnerabilities the whole world already knows about. Athena adds a second, very different stream. Coalition members share zero-day vulnerabilities that haven't been disclosed yet. Chainguard remediates those vulnerabilities, and we deliver those fixes to members under embargo, usually 30 days, before the vulnerability becomes public.

Those two streams can't be kept in separate corners, because they impact the same packages with different confidentiality rules. Take a customer running a version we've already patched for a disclosed CVE who then needs the embargoed fix for a zero-day in that same version. They need both. If the zero-day fix didn't sit on top of the CVE fix, we'd be handing them protection against tomorrow's threat while stripping their protection against yesterday's. So both fixes have to live on the same package at once.

So a single package now carries two kinds of patches with opposite requirements. The disclosed CVE fixes have nothing to hide; we want them to be openly attributable. The Athena fixes must be provably impossible to leak: the patch, and sometimes the fact that a patch exists at all, has to stay invisible until disclosure. And the private fixes must stay stacked on the public ones, permanently, as both keep moving.

Historically, you’d use Git to solve this problem with branches. However, git doesn't give you a confidentiality boundary between two histories in the same repository. It also won't keep one history continuously restacked on another as upstream shifts beneath both.

So we built something that does. It's called forkstore, and it holds those two lines of remediations together with the necessary confidentiality boundaries to prevent them from ever mixing before a given vulnerability is disclosed.

One fork, many lines

forkstore keeps one bare git repository (one fork) per package. Each fork is a real git repository even when the upstream doesn’t use git. We convert Subversion, Mercurial, and the rest into git at fork time, so everything downstream sees one consistent interface. History inside a fork is organized around a concept we call a line.

A line works like a git branch, a named pointer to a sequence of commits, except that it always begins at a pristine upstream release and only ever grows by stacking our remediations on top. The bottom commit is the anchor: the exact upstream commit for that release, and nothing of ours. (Working out which upstream commit actually corresponds to a released version is itself surprisingly hard; that's a post for another day.) Everything above the anchor is the work Chainguard adds: first, the source changes that make the package buildable in our system, then the security patches, one commit each. A release is just a tag on that stack. Because the stack sits directly on the pristine anchor, the diff between a release tag and the anchor is exactly the patch set we shipped. "What did Chainguard change here?" has a precise, mechanical answer.

Each commit records what it does, too. A backport names the CVE it fixes in a structured trailer, so a line's history is a self-describing record we can generate the public advisory feed straight from.

Here's a real line, for Apache HttpComponents Core 5.4.2:

public/5.4.2   (anchored to upstream rel/v5.4.2)
  ├─ backport fix for CVE-2026-54428      ← tag: cgr/5.4.2-0.cgr.1
  ├─ backport fix for CVE-2026-54399
  ├─ build enablement for httpcore5@5.4.2  ← baseline pin (cgbase tag)
  └─ anchor — upstream release rel/v5.4.2  (pristine)

There are three commits above a pristine anchor: one to make the package buildable, then the two backported CVE fixes. The top is tagged 5.4.2-0.cgr.1, the first public remediated release of this version, which bundles both fixes into a single artifact. As we backport more fixes for 5.4.2, they stack on this same line and ship as the next cgr release.

The public–private split

forkstore exists to reconcile public and private vulnerability streams. We built the split into the storage layer itself rather than bolting it on as access control. A boundary enforced by where an object physically lives can't be misconfigured the way a permission can.

Disclosed-CVE backports are cut as cgr.N releases in the public layer; undisclosed Athena remediations as cgp.N releases that only ever exist in the encrypted one. Same package and machinery; two stores that can never be confused for one another.

A single package's objects live in layered stores:

• A base layer holding a read-only cache of upstream history, written once and never amended again

• A public layer holding everything we assert publicly: the backport releases that fix publicly disclosed CVEs (the cgr tags), and the build-enablement baselines (the cgbase tags)

• A private layer, in a separate encrypted store, that holds confidential vulnerabilities coalition members submitted to Athena: these are backported releases that fix undisclosed findings (the cgp tags), which can't exist anywhere public until the embargo lifts.

We use git's alternates mechanism so the private layer can resolve objects from the public and base layers beneath it. A private line builds directly on public history, while the public layer has no knowledge that the private layer exists. Confidentiality becomes a property of where an object physically lives. A private commit can't leak into a public artifact, because the machinery that builds public artifacts is wired to a store that doesn't contain it. That makes "someone misconfigured an ACL and an embargoed fix showed up in a public feed" structurally impossible.

Keeping the two in lockstep

Aside from keeping private lines separate from public lines, the other thing forkstore has to do is keep both public and private lines in sync. This is because the public base a private line is stacked on is a moving target based on the number of CVEs Chainguard has remediated.

And this base moves constantly. We are always remediating newly disclosed CVEs in a given version, so public lines keep gaining backports, and each one advances the base that the Athena patches sit on. A private line is, by deliberate definition, the public line plus a stack of private-only commits. That's what guarantees an embargoed build always includes every public fix beneath it, so a customer never has to choose between "has the zero-day fix" and "has last week's CVE fixes." But the instant a public backport lands, every private line on that version is stale.

So reconciliation runs continuously. When a public head moves, every private line based on it is enqueued and restacked onto the new head. Public content flows up into private work; nothing private ever flows down. The resting state is "every private line is stacked on the latest public head," and forkstore is always driving toward it, so a line never gets stuck half-restacked even if a reconcile fails partway.

Public and private fixes tend to touch the same code, so re-stacking throws merge conflicts more often than not, and doing it by hand across thousands of lines every time a base moves can’t scale. The restack is agentic. An agent resolves the conflicts, then rebuilds the package and runs its tests to confirm it still works before the reconciled line is allowed to stand. Automating the mechanical part of a restack is easy; automating the judgment the conflicts demand is what makes continuous reconciliation actually continuous.

Declassification without deletion

The payoff of all this shows up when an embargo lifts and a private fix becomes public. You might expect that to mean carefully moving a commit from the encrypted store to the public one, which you’d never want near a confidentiality boundary. forkstore moves nothing. Ever. Declassification is a human-approved action that resubmits the fix as an ordinary public landing, through the same path as any other public patch. Then, on the private line's next restack onto the advanced public head, the reconciler finds its commit already present in the public base and absorbs it. There is no code path that deletes a private commit. The private copy is made redundant and falls away. A boundary you never have to delete across has far fewer ways to go wrong.

Building the foundation of Athena

Version control is not glamorous work, but we built a new version for holding public, auditable CVE fixes and secret, embargoed fixes on the same package versions, something git was not designed to do. Our design ensures private and public lines never intersect, and keeping private lines stacked on the public base keeps getting safer with more backported fixes. forkstore is the foundation on which the rest of Athena's remediation stands, and the confidentiality guarantee lives in the storage layer itself rather than in anyone's diligence. When Athena disclosed its first vulnerabilities, the remediated artifacts customers pulled that day were built from these lines.

Check out Athena to learn more about how all this helps protect open source software from AI attacks.