Why your company is wasting thousands of hours on software vulnerabilities

TL;DR Software companies that build and operate containers are spending thousands of hours each year, if not more, on vulnerability management. Why? According to approximately ten interviews with software professionals, it’s because software companies are often swamped with thousands of known software vulnerabilities (Common Vulnerabilities and Exposures, aka CVEs), a result that stems in part from some software teams selecting images regardless of the count of known vulnerabilities.

‍

If you’re a software company that builds and deploys containers, it’s possible, even likely according to our research, that there are thousands of unique known vulnerabilities (CVEs) across your software products. This is not just a security risk. It’s also possible that the burden of known vulnerabilities is turning the daily routine of the software professionals at your company from one of joy in shipping new features and watching customers benefit to a slog involving the identification, triage, and remediation (“management”) of known software vulnerabilities. This slog becomes a black hole for productivity, as team members are forced to chase down vulnerabilities across software languages, apps, and more avenues that require specialized knowledge to remediate.

‍

To better understand the magnitude and origins of this problem, Chainguard Labs interviewed approximately ten software professionals that conduct vulnerability management across a range of companies that build or deploy containers. Key findings include:

‍

Based on these interviews, it’s likely that many companies spend thousands of hours or more on vulnerability management each year. This translates into the work of at least a couple full time staff members per year.

• This time burden results from there being thousands of unique known vulnerabilities across the containers built or operated by modern software companies. Though there are many reasons for this, a key cause is that some software development teams pick images without regard to the known vulnerability count, inadvertently imposing costs on other teams throughout their company.

‍

Read on to dive deeper into the findings and learn how Chainguard Images, container images with low-to- zero known vulnerabilities, can dramatically reduce the time burden associated with vulnerability management.

‍

How much staff time do companies that build and ship container-based software spend on vulnerability management?

The interview research revealed that crude top-level estimates of the total amount of time spent by software professionals directly triaging and remediating vulnerabilities are possible, at least approximately. Staff were often able to estimate the time spent on these activities by themselves and closely related colleagues and then to extrapolate.

‍

See the table below for approximate estimates of the total time spent directly by staff on vulnerability management. It’s worth noting that these interviews focused on estimates of day-to-day vulnerability management and not vulnerability management during periods of crisis like log4shell. The table is arranged from organizations estimated to be spending the most hours on vulnerability management to the least.

So what’s the takeaway from this table? Some companies spend thousands of hours or more on vulnerability management each year. But not all.

‍

Nonetheless, some companies and organizations are dealing with an immense amount of vulnerability management. One transport and logistics company spends the equivalent of ten person-years on vulnerability management each year. Another organization, with federal ties, also spends an extraordinary amount of staff time on vulnerability management.

‍

Why do companies that build and ship container-based software spend so much time on vulnerability management?

‍The interview results pointed to a number of reasons. This list describes the two most significant factors, according to these interviews, why some software companies spend so much time managing vulnerabilities in containers.

‍

First, developers pick base images (and other images) without regard to the number of known software vulnerabilities. In the language of economics, this creates “negative externalities” for other roles in a software company. Developers sometimes (often, in fact, according to platform engineers) pay little heed to the number of known vulnerabilities in base images.

‍

Second, CVE remediation time depends on the ease of upgrading and testing software. For organizations that have low test coverage, require extensive manual testing, slow builds, or very outdated dependencies, the burden of fixing vulnerabilities, even via minor version bumps of a dependency, is high. Think of this as the “DevOps” view of vulnerability management. If your company struggles at DevOps (and the majority do), vulnerability management is a particularly big pain.

‍

The best vulnerability management is less vulnerability management

‍In short, managing known software vulnerabilities has become a major time burden at modern software companies that build or operate containers. And this situation appears to result from a “negative externality” in which some software teams are unaware of the known software vulnerability burden they impose on other teams.

‍

While some parties advocate for sophisticated vulnerability analysis and enrichment, there’s another option: ship software with zero-known vulnerabilities. That’s what Chainguard Images offers. These are images with low-to-zero known vulnerabilities powered by Wolfi. By radically reducing vulnerabilities across a company’s containers, it’s possible to recoup thousands of hours otherwise allocated to vulnerability management.

‍

Read the full report to learn more about how companies are approaching vulnerability management, or reach out to our team to explore how Chainguard Images can help you save time and increase productivity.