Into the deep: Exploring Chainguard Container Images

At Chainguard, we focus a lot on the value our Images product brings to our customers via our low-to-no Common Vulnerabilities and Exposures (CVEs) commitment, but the reality is that the value of Chainguard Images to our customers’ supply chain security extends far deeper than just low CVE counts. As one person put it at our recent quarterly business review: Come for the low CVEs, stay for the supply chain security. That is exactly right.

‍

Low-to-no CVEs

Virtually every compliance framework I have seen has a section on “Vulnerability Management.” It appears in SOC2, FedRAMP, PCI DSS, ISO27001, and more. The strength of the requirements are the main piece that varies, and it ranges from “track them, and have a plan” to “eliminate them above a certain severity, within a specific timeframe.”

‍

For example, PCI compliance historically requires users to fix any Critical or High CVEs uncovered by network scans and rescan. However, even just tracking CVEs (the bare minimum) via the infamous “CVE spreadsheet” is an enormous source of toil for developers. That toil translates directly to costs, as developers are often one of the most expensive assets a business has.

‍

At Chainguard, we are at the forefront of solving the CVE problem with our Wolfi un-distro, which is able to pick up CVE patches, apply them to packages, and get them to customers extremely quickly. This is so fundamental to the value our customers get that we offer SLAs around remediation in our enterprise product.

‍

We lead with this messaging (the tip of our messaging “iceberg”) because complying with these sections on “vulnerability management” is such a universal source of pain (and cost) for our customers today. However, in addition to solving our customers “today problems,” we are also anticipating their “tomorrow problems” with respect to supply chain security.

‍

Minimal Images

A growing best practice, which is also starting to appear in the compliance frameworks, is to utilize minimal images (see also). For example, the September 2022 update from the PCI council with guidance for containers states:

‍

Threat: Container base images downloaded from untrusted sources, or which contain unnecessary packages, increase the risk of supply chain attacks.

‍

Best Practice: Application container images should be built from trusted, up-to-date minimal base images.

‍

At Chainguard, we are pioneers at the forefront of minimal images. Dan Lorenc and I created the original “distroless” project as a prototype to prove the concept in our time at Google. With Ariadne Conill and team, we have taken this to new levels at Chainguard, with the only purpose-built tooling to do this sustainably at scale.

‍

A key benefit of minimal images, which we reported on previously, is that the smaller surface can lead to a substantial reduction in the rate at which images accumulate CVEs (this ties into pain of the previous section). We also see minimal images as a key mitigation for the growing evasion technique known as “living off the land,” which attackers are using to avoid intrusion detection software. Another benefit of minimal images is that they are small (comparing a crude sampling with official images, the average is 79% smaller).

‍

Chainguard’s Images are minimal by default, but many contain what we call a “dev variant” to enable a smoother migration path to distroless nirvana.

‍

Signing

Another growing best practice is to require images to be signed by authorized build pipelines. This is also starting to appear in the compliance frameworks. For example, the September 2022 update from the PCI council with guidance for containers states:

‍

Threat: Known good images can be maliciously or inadvertently substituted or modified and deployed to container runtime environments.

‍

‍Best Practice: Registries should be configured to integrate with the image build processes such that only signed images from authorized build pipelines are available for deployment to container runtime environments.

‍

At Chainguard, we are also pioneers at the forefront of software signing. Dan Lorenc and Kim Lewandowski helped create the Sigstore project and OpenSSF. Dan also invented Sigstore’s keyless signing, and worked with Jon Johnson to create the first prototype of Sigstore’s cosign project. Billy Lynch created the initial implementation of Sigstore's gitsign project. Priya Wadhwa sits on the Technical Steering Committee (TSC), and many of us continue to help support Sigstore in a wide variety of functions (such as on-call for the public service).

‍

Software Bill of Materials (aka SBOMs)

In 2021, the White House dropped a BOMshell (sorry, I couldn’t resist) with Executive Order 14028; which called for:

‍

(vii) providing a purchaser a Software Bill of Materials (SBOM) for each product directly or by publishing it on a public website

‍

The Chainguard team has been engaged in the discourse around SBOMs from the beginning (before we were Chainguard) with NTIA and then with CISA. Adolfo Veytia, one of the foremost experts in SBOM, authored the Kubernetes project’s bom tool, and drove the Kubernetes Enhancement Proposal (KEP) to publish SBOMs for the ubiquitous Kubernetes project. Earlier this year, the DHS awarded Chainguard a grant to lead the development of SBOM composition tooling, which resulted in the protobom tool.

‍

Many folks describe SBOMs as a “list of ingredients” for the image, describing each of the pieces that went in (e.g. package and version, source and commit, etc).

‍

All Chainguard Images contain SBOM attestations, which are signed by our release automation, so that processes consuming the SBOMs can validate the authenticity of our SBOM’s provenance. As the compliance landscape continues to evolve and come into focus, we continue to iterate on the shape of our SBOMs to ensure that they are compliant with SBOM regulations as soon as they land (such as the NTIA “minimum elements”).

‍

SLSA (Supply-chain Levels for Software Artifacts)

In 2021, Google released SLSA as an open specification to help users secure parts of their supply chain. It is based around the ideas in “Binary Authorization for Borg” developed and implemented by Google for a number of years as a way of securing its internal production environment known as Borg, which itself inspired and informed the creation of the open source project Kubernetes.

‍

At Chainguard, we are big advocates of SLSA. Kim Lewandowski helped create and launch SLSA. Several of us participated in the early working groups, and had a deep background in its predecessor from our time at Google. Chainguard has also performed a number of SLSA audits for our customers and organizations like the CNCF.

‍

One of the key elements of SLSA is the provenance attestation, which has some overlap with SBOM. But generally, if an SBOM is the “list of ingredients,” you can think of the provenance attestation as the recipe (how those ingredients combined to produce the image).

‍

All Chainguard Images carry SLSA attestations, which are signed by our release automation, so that processes consuming the attestations can validate the authenticity of our SLSA provenance. We have tracked SLSA from v0.1 through to v1.0.

‍

Reproducibility

Reproducibility has not yet found its way into the requirements of any compliance frameworks, but it was briefly one of the aspirational elements of SLSA level 4 (which was left out of v1.0). It is also increasingly a focus of the supply-chain inclined.

‍

At Chainguard, we see reproducible builds as a cheat code for supply chain security. When a build is reproducible, then its provenance becomes auditable because you can rerun the build to verify the result! We have even extended the SOURCE_DATE_EPOCHconcept to a new concept we call build date epoch.

‍

All Chainguard Image builds are reproducible, which we verify before ever tagging a new image and distributing it to our users.

‍

General Hardening

The list goes on and on. At Chainguard, we work across every single stage of our pipeline to harden toolchains, applications, packages, and images. Some of these things (like defaulting to a nonrootuser) have made their way into compliance frameworks like the Sept 2022 update from the PCI council:

‍

Threat: The default position of Linux containers, which is to run as root, could increase the risk of a container breakout.

‍

‍Best Practice: Container images should be built to run as a standard (non-root) user.

‍

However, many of the things we are doing go above and beyond the current compliance standards.

‍

At the toolchain level, we are adopting more secure default flags when building applications. For example, we default to FORTIFY_SOURCE=3for C++. In Go, we default to disabling tarinsecurepathand zipinsecurepath, which were introduced as options in Go 1.20; we are also considering things like defaulting to -mode=pie.

‍

At the filesystem level, we have embraced the Filesystem Hierarchy Standard (FHS).

‍

At the distro level, we embrace the use of memory safe languages for critical libraries and utilities (e.g. rustls, and sudo-rs).

‍

If you have ideas for ways that we can continue to raise this bar, please reach out! We are always looking for more ways to improve.

‍

Future: Vulnerability Exploitability eXchange (VEX)

VEX is the lesser known sibling of SBOM, and is another format being pushed by CISA. It is intended to be a way of asserting that a particular vulnerability does not apply to a particular software artifact (aka “a false positive”). It is still early days for VEX, and so it has not yet appeared in any compliance standards.

‍

Adolfo Veytia, Dan Luhring, and the Chainguard team have been working to advance the state of VEX releasing the OpenVEX specification and the vexctl CLI (now under OpenSSF). We have also partnered with Anchore to bring OpenVEX support to grype as a reference for the vulnerability scanner ecosystem.

‍

At Chainguard, we see the OpenVEX specification as a way of eliminating false positives from scanners with fuzzy matching logic on container images, including, of course, Chainguard Images!

‍

One of the projects the team is now exploring is how we can surface the Wolfi advisory data as OpenVEX statements, so that we can expand our scanner integration guide and scanners like grypecan better take advantage of our advisory feed.

‍

Come for the low CVEs, stay for the supply chain security

At Chainguard, we are on a mission to be the safe source for open source. To become that we are building a holistic solution to software supply chain security with, or as the CNCF TAG Security calls it: a secure software factory (contributions by: Dan Lorenc, Jason Hall, Priya Wadhwa, Dan "POP" Papandrea, and Matthew Moore).

‍

We are always working to ensure that our Images meet today’s compliance standards, but we are also anticipating (and sometimes helping shape) future compliance standards. However, our efforts also go well beyond the bare minimum for compliance, and we are constantly searching for ways to raise the bar across our catalog to adhere to (or even create) new best practices for software supply chain security.

‍

Nobody competes with our low-to-no CVE commitment today, and as you can see even that is just the “tip of the iceberg” of Chainguard Images’ value. Reach out if Chainguard can help you achieve your compliance and software supply chain security goals.