Distroless images are a fantastic way to shrink your container attack surface, but they come with a hidden cost: no shell means you can’t just exec into them to poke around. Trivy, the popular vulnerability scanner, can still find vulnerabilities in these stripped-down images, but understanding how it does it is key to trusting its output and using it effectively.
Here’s Trivy scanning a distroless Go application image:
$ trivy image --format template --template '@/contrib/template/common/vuln.tpl' --severity HIGH,CRITICAL ghcr.io/aquasecurity/trivy:latest
Output:
{
"Results": [
{
"Target": "ghcr.io/aquasecurity/trivy:latest (linux/amd64)",
"Class": "language-package",
"Type": "apk",
"Vulnerabilities": [
{
"VulnerabilityID": "CVE-2023-39325",
"PkgName": "openssl",
"InstalledVersion": "1.1.1w-r0",
"FixedVersion": "1.1.1w-r1",
"Severity": "HIGH",
"Title": "openssl: Privilege escalation vulnerability in the X.509 certificate parsing",
"Description": "A vulnerability in the X.509 certificate parsing in OpenSSL versions prior to 1.1.1w, 3.0.x prior to 3.0.11, and 3.1.x prior to 3.1.4 allows an attacker to cause a denial of service via a malformed certificate.\n\n\nThis issue was addressed by updating OpenSSL to version 1.1.1w, 3.0.11, and 3.1.4. This fix is being backported to supported stable releases.\n\n\nCVE-2023-39325 was reported by Tomislav Mamic and Matt Caswell from the OpenSSL project.",
"References": [
"https://github.com/openssl/openssl/pull/21688",
"https://www.openssl.org/news/secadv/20230801.txt"
],
"PublishedDate": "2023-08-01T04:15:00Z",
"LastModifiedDate": "2023-08-10T07:05:00Z",
"CVSS": {
"3": {
"Vector": "CVSS:3.1/AV:N/AC:L/PR:N/UI:N/S:U/C:N/I:N/A:H",
"Score": 7.5
},
"2": {
"Vector": "AV:N/AC:L/Au:N/C:N/I:N/A:P",
"Score": 5.0
}
},
"Target": "openssl",
"Remediation": {
"Fix": "openssl=1.1.1w-r1"
}
}
]
},
{
"Target": "ghcr.io/aquasecurity/trivy:latest (linux/amd64)",
"Class": "os-package",
"Type": "dpkg",
"Vulnerabilities": []
}
]
}
When you scan a standard image, Trivy often uses docker exec or similar mechanisms to get a list of installed packages and their versions. It then compares these against its vulnerability database. With distroless images, this direct approach is impossible because there’s no shell, no /bin/sh, and often no package manager like apt or apk installed.
Trivy sidesteps this limitation by leveraging the image’s filesystem. Instead of executing commands inside the container, it analyzes the files within the image layers. For OS packages (like Debian’s .deb or Alpine’s .apk), Trivy looks for package metadata files that are still present in the filesystem, even without the package manager binaries. For language-specific packages (like Python’s site-packages or Node.js’s node_modules), it parses manifest files such as requirements.txt, package.json, or even compiled dependency information.
Let’s dive into how it works for different package types:
OS Packages (e.g., Alpine APKs, Debian DEBs):
Distroless images often still contain the actual package files and their metadata. Trivy can scan these directly.
- Diagnosis: Trivy will identify the package manager type (e.g.,
apk,dpkg). It then searches the image filesystem for known locations of package databases. For Alpine, this might be/lib/apk/db/. For Debian, it could be/var/lib/dpkg/. - Mechanism: Trivy reads the package database files (e.g.,
world,installedforapk;status,infofordpkg) that are still present in the image’s filesystem. These files contain the package names and installed versions. Trivy then cross-references these with its vulnerability database, just as it would with a live system. - Fix: Update the base image to a newer version that includes patched packages. For example, if your distroless image is based on
alpine:3.18and Trivy reports anopensslvulnerability fixed in1.1.1w-r1, you’d rebuild your image using a base that hasopensslat or above that version. This is typically done by updating theFROMinstruction in your Dockerfile.
Language-Specific Packages (e.g., Python, Node.js, Go):
This is where distroless images really shine, and where Trivy’s analysis becomes crucial. Trivy doesn’t need a runtime to find vulnerabilities in your application’s dependencies.
- Diagnosis: Trivy identifies the programming language and searches for common dependency manifest files (
requirements.txt,Pipfile,pyproject.tomlfor Python;package.json,package-lock.json,yarn.lockfor Node.js;go.mod,go.sumfor Go). It also looks for installed packages in language-specific directories (e.g.,/usr/local/lib/python3.11/site-packages/,/app/node_modules/). - Mechanism: For interpreted languages like Python and Node.js, Trivy parses the manifest files to get a list of direct and transitive dependencies and their versions. It then queries its vulnerability database for known issues in those specific versions. For compiled languages like Go, it can often parse the compiled binary or the build artifacts to infer dependencies and versions, or directly read
go.mod/go.sum. - Fix: Update the specific dependency in your application’s manifest file to a version that is not vulnerable. For example, if
package.jsonlistslodashversion4.17.10and Trivy flags a vulnerability, you would change it tolodash: "^4.17.21"(or another patched version) and then rebuild your distroless image.npm installoryarn install(done during the build process, not in a runtime container) will fetch the new version, and Trivy will rescan the updatednode_modulesor manifest.
Example: Scanning a Go Distroless Image
Consider a simple Go application Dockerfile:
FROM golang:1.20-alpine AS builder
WORKDIR /app
COPY . .
RUN go mod download
RUN CGO_ENABLED=0 GOOS=linux go build -o /app/my-app .
FROM gcr.io/distroless/static-debian11
COPY --from=builder /app/my-app /app/my-app
COPY --from=builder /etc/ssl/certs/ca-certificates.crt /etc/ssl/certs/
ENTRYPOINT ["/app/my-app"]
When Trivy scans my-image:latest (built from this Dockerfile), it won’t find any OS packages to report on because distroless/static-debian11 is extremely minimal. However, it will still analyze the Go binary (/app/my-app) if it has Go support enabled.
- Diagnosis: Trivy detects it’s a Go binary.
- Mechanism: Trivy can analyze the Go binary’s embedded metadata or look for
go.mod/go.sumfiles if they were copied into the final image (though they usually aren’t in minimal distroless builds). More sophisticated analysis might involve disassembling parts of the binary to identify imported libraries and their versions. It relies heavily on its vulnerability database that maps Go module versions to CVEs. - Fix: Update your Go modules using
go get -u <module_path>or by manually editinggo.modto a non-vulnerable version, then rebuild the image.
The key takeaway is that Trivy’s strength lies in its ability to inspect the contents of an image’s filesystem without needing to execute anything within it. This makes it perfectly suited for scanning distroless images, as it can find vulnerabilities in OS packages and application dependencies by directly analyzing the files that make up the image.
The next hurdle you’ll likely encounter with distroless images is debugging runtime issues, as the lack of a shell makes traditional troubleshooting impossible.