The incremental path to container images: Chainguard Images

The most important value proposition of Chainguard Images that we hear from our customers and prospects is our low-to-no Common Vulnerabilities and Exposures (CVEs) commitment for every container image. It enables them to eliminate the toil their developers feel from managing the “CVE spreadsheet" and also enables them to sail through the vulnerability management portions of compliance frameworks like FedRAMP, SOC2, ISO27001, PCI DSS, and more.

‍

For some customers and organizations, compliance with these frameworks is make or break for hundreds of millions of dollars of revenue. However, migrating from existing images to Chainguard’s hardened Images can seem like a lot of scary work. To make things easier, I wanted to break it down in ways that enable users to adopt our Images quickly and effectively.

‍

One of my first rules of “code reviews” (and life) is to separate out contentious things from non-contentious things. Don’t link your ability to land something non-contentious to something contentious that may take time to resolve! Let’s apply a few iterations of this rule to adopting Chainguard Images, and see how we do.

‍

Starting with Application Images

The easiest Chainguard Images for users to adopt are our Application Images (vs. our Base Images). These Images are our hardened, drop-in replacements for various upstream projects, where users can typically change which images are used by a Helm chart (or other manifests) to pull Chainguard Images instead.

‍

These applications can range in complexity from a single image (e.g. Postgres, Redis, Prometheus), to a few images (e.g. Cert Manager, KEDA, Kyverno), to many images (e.g. a full Sigstore stack). Generally, users run them without needing to do any modification or customization of the image itself.

‍

In the wonderful cloud-native world of Kubernetes, these components often comprise a non-trivial number of the images that wind up running in users’ environments, too. This is often the “lowest hanging fruit” for reducing CVEs through Chainguard Images.

‍

For example, consider that Istio’s proxy image runs in every single pod participating in the service mesh. It’s often what exposes users’ clusters to the outside world via its gateways. This is definitely something you want to be hardened, and something that is extremely unlikely to need modifications from end users.

‍

Base Image Baby Steps

Once we have tackled all the Application Images that users can just take and run “off-the-shelf,” we are left with the images that they are building themselves. Images that users build themselves generally layer on top of some form of “base” image, most often pulled in via a line like:

FROM alpine:3.18

This is often a place in the adoption of Chainguard Images where scary words like distroless can send users running for the hills!

This is a common misconception! It’s not actually the exclusion of packages that give Chainguard Images low-to-no CVEs; it is the speed with which we apply patches to software and include it in our Images. Going distroless can reduce CVEs if the software you are excluding is unpatched, but in Wolfi we patch everything. As a result of this, our traditional “distroful” images typically have the same low-to-no CVE counts as our “distroless” images.

‍

This brings us to our next contentious vs. non-contentious split: for the next step in our journey, let’s put a pin in going distroless.

‍

But How?

While we strive to make our default posture at Chainguard the most secure (a la distroless), we also understand that users need a path to get there, or it is all for naught. This is one of the reasons that virtually every Chainguard Image comes with what we call a “dev variant.” These “dev variants” include all the goodies that users would generally expect from their traditional base images, which are also typically the things that are absent from distroless images, such as a package manager and shell.

‍

The presence of these tools make our “dev variants” much closer to a drop-in replacement than our default distroless images. Here are some examples:

‍

‍

For some images, we don’t publish a dev variant simply because the image itself is already “distroful”. An example of this would be our chainguard-baseimage (this is wolfi-basein our free tier), which can often be used as a replacement for simple distro-based images, like our original example:

‍

‍

Similar to Alpine, Chainguard’s Wolfi-based Images utilize the apk package manager, so users looking to install additional packages can do so with the familiar command:

RUN apk add curl

For users of deb-based based images (e.g. Ubuntu, Debian), this would replace aptcommands. For users of rpm-based images (e.g. Redhat, Fedora, CentOS, Suse, Amazon Linux), this would replace yumcommands. In all these cases, you may see variations in package names, but as a pro tip, you can search for which packages provide particular commands via:

# apk search cmd:curl
curl-8.4.0-r1

Wolfi already has a large number of popular packages built, but if you find gaps, then you can file a request (or add it) on the Wolfi GitHub repo. Customers can also reach out to us via support channels to help us prioritize requests for missing packages.

‍

Once you have adopted Chainguard for all of your Base Images (as well as your Application Images), then you will start reaping the benefits of our key value proposition of low-to-no CVEs across all of your images.

‍

If achieving low-to-no CVEs was all that you wanted, then congratulations, you can stop here. However, if you’re interested in the additional benefits, then let’s continue on our journey.

‍

Chasing Distroless

While “distroless” is not a requirement for low-to-no CVEs, it is a very complementary layer of a defense-in-depth strategy. In particular, distroless-style images accumulate new CVEs 5-6x more slowly than their traditional counterparts on average. This means that users need to pick up updates to Application and Base Images, and roll them out to production much less frequently.

‍

Another key benefit of the distroless-style images is that by excluding the tools that cater to human operators (like a package manager and shell) you mitigate an evasion technique that attackers use to avoid intrusion detection called “living off the land,” where they take advantage of these and other tools that have been left in the runtime environment.

‍

These days, there are a number of techniques for building application images leveraging a distroless Base Image including:

• ko (for Go)

• Jib (for Java)

• Multi-stage Dockerfile

• Bazel's rules_oci and rules_apko

If you are following the incremental steps of this journey, then in all likelihood a Multi-stage Dockerfile is the right next step for you, but I would highly recommend looking into some of the other options listed here (out of scope for this adventure), which form a class I call “last-mile” image builders.

‍

Generally, the final structure you will find for distroless multi-stage Dockerfiles will look something like this (note the absence of RUNdirectives after the final FROMsince you cannot RUNin a distroless image):

FROM ... AS builder

# build the application

FROM cgr.dev/my.co/a-runtime-image
COPY --from=builder path/to/build/output /where/you/want/it
ADD config/files /somewhere/else
ENTRYPOINT ["/where/you/want/it"]

However, this is the goal state; it is not where you start! The first steps on your way to this would be to start restructuring your application into the builder / runtime environment pattern like this:

FROM ... AS builder

# build the application

FROM cgr.dev/my.co/something:latest-dev     # the closest "dev variant"
RUN apk add some set of packages
COPY --from=builder path/to/build/output /where/you/want/it
ADD config/files /somewhere/else
ENTRYPOINT ["/where/you/want/it"]

Once you have gotten your multi-stage Dockerfile into roughly this shape, you really want to scrutinize the necessity of the packages you are adding via RUN apk add ..., and, true to the distroless philosophy, remove them wherever possible. In a perfect world, they will all go away and you can drop the -devsuffix on the Chainguard Base Image. However, in certain cases we have customers with special runtime requirements. For those cases we can produce a distroless Custom Image for the customer that extends the Base Image from our catalog with a custom package list. We just need to know these two lines:

FROM cgr.dev/my.co/something:latest-dev
RUN apk add some set of packages

Armed with this information, we are able to craft a variant of the Base Image that include the additional packages, and exposes it to the customer’s cgr.dev/my.coOCI repository so they can drop the -devsuffix and achieve distroless nirvana:

FROM ... AS builder

# build the application

FROM cgr.dev/my.co/custom-something
COPY --from=builder path/to/build/output /where/you/want/it
ADD config/files /somewhere/else
ENTRYPOINT ["/where/you/want/it"]

That's a Wrap

Hopefully you will find this segmentation of the journey useful, and it helps your organization to reach its goals, whether that is simply CVE reduction, compliance with regulations, or an adherence to the best practices of the distroless philosophy across all of your images. Combining all these practices maximizes software supply chain security and keeps your customers and data safe.

‍

If your organization has any of these goals in mind, then Chainguard is here to help. Contact us now to get started.