The Vault Agent Sidecar is the most elegant way to get secrets into Kubernetes pods, but it’s not the only way, and understanding its mechanics unlocks a deeper appreciation for how Kubernetes itself handles secrets.
Let’s see it in action. Imagine you have a Kubernetes Deployment that needs a database password stored in Vault.
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 1
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: app-container
image: nginx:latest # Replace with your actual app image
ports:
- containerPort: 80
volumeMounts:
- name: vault-secrets
mountPath: "/etc/secrets"
readOnly: true
volumes:
- name: vault-secrets
emptyDir: {} # This is where the secrets will be mounted
This is your basic app Deployment. It defines an emptyDir volume named vault-secrets and mounts it into the app-container at /etc/secrets. Now, we need to inject the Vault Agent Sidecar to populate this volume.
Here’s the crucial part: the Vault Agent Sidecar isn’t directly injecting secrets into the application container. Instead, it’s acting as a local Vault client that watches Vault for changes and writes secrets to a shared volume. The application container then simply reads from this shared volume.
apiVersion: apps/v1
kind: Deployment
metadata:
name: my-app
spec:
replicas: 1
selector:
matchLabels:
app: my-app
template:
metadata:
labels:
app: my-app
spec:
containers:
- name: app-container
image: nginx:latest # Replace with your actual app image
ports:
- containerPort: 80
volumeMounts:
- name: vault-secrets
mountPath: "/etc/secrets"
readOnly: true
# --- Vault Agent Sidecar ---
- name: vault-agent-sidecar
image: hashicorp/vault:latest # Use a specific version for production
args:
- "agent"
- "-config=/etc/vault/agent-config.hcl"
volumeMounts:
- name: vault-secrets
mountPath: "/vault/secrets" # Where the agent writes secrets
- name: vault-agent-config
mountPath: "/etc/vault"
# --- Volumes ---
volumes:
- name: vault-secrets
emptyDir: {} # Shared volume for secrets
- name: vault-agent-config
secret:
secretName: vault-agent-config-secret # Kubernetes Secret holding the agent config
The vault-agent-sidecar container uses the hashicorp/vault:latest image and runs the vault agent command. It’s configured via an agent-config.hcl file, which we’ll store in a Kubernetes Secret named vault-agent-config-secret. This agent is configured to write secrets to /vault/secrets within its own container. Crucially, the vault-secrets emptyDir volume is mounted into both the app-container (at /etc/secrets) and the vault-agent-sidecar container (at /vault/secrets). This shared volume is the key to the sidecar pattern.
Here’s a sample agent-config.hcl:
# /etc/vault/agent-config.hcl
log_level = "info"
api_addr = "http://vault.default.svc.cluster.local:8200" # Your Vault address
auto_auth {
method {
type = "kubernetes"
config = {
role = "my-app-role"
}
}
}
listener "tcp" {
address = "127.0.0.1:8181"
tls_disable = true
}
# This section tells the agent to sync secrets to disk
# The path here is relative to the agent's container's filesystem
# We want to sync to /vault/secrets, so we map it here.
# The `target` path in Vault is where your secret is stored.
# The `destination` path is where the agent writes it on disk.
# The `mode` can be "file" or "template"
# In this case, we want to write the full secret content to a file.
# The file will be named `database.txt` and placed in `/vault/secrets/`
# which is mounted to the application container.
# The content of the file will be the value of `database/config/creds/my-app`
# from the KV v2 store.
sync {
secret {
path = "secret/data/database/config/creds/my-app" # KV v2 path
destination = "/vault/secrets/database.txt" # Where the agent writes it
mode = "file"
}
}
The auto_auth block uses the Kubernetes auth method to authenticate with Vault. The role specified here must be configured in Vault to allow pods with a matching Kubernetes Service Account to authenticate. The sync block is where the magic happens: it tells the Vault Agent to watch a specific secret path (secret/data/database/config/creds/my-app for KV v2) and write its content to a file (/vault/secrets/database.txt).
When the pod starts, the Vault Agent authenticates with Vault using its Kubernetes Service Account identity. It then starts watching the configured secret path. Once the secret is available, the agent writes it to the shared vault-secrets volume as database.txt. The app-container can then read this file at /etc/secrets/database.txt.
The most surprising true thing about this pattern is that the Vault Agent Sidecar doesn’t actually need to be a sidecar in the traditional sense. It could be a separate process running on the node, or even a standalone DaemonSet. The sidecar pattern is primarily an organizational choice that keeps the secret management logic colocated with the application it serves, simplifying the Kubernetes manifest and ensuring that the secrets are available before the application container is fully ready.
The true power here is the separation of concerns. Vault handles secret storage and lifecycle. The Kubernetes Secret object (holding the agent config) manages how the agent is instructed. The shared emptyDir volume provides the transport mechanism. And the application container simply consumes its dependencies from a file.
One thing people often overlook is how the sync block’s destination path interacts with the volumeMounts. The destination path is relative to the agent container’s filesystem. Because the vault-secrets volume is mounted at /vault/secrets in the agent container and /etc/secrets in the app container, writing to /vault/secrets/database.txt by the agent makes that file accessible as /etc/secrets/database.txt to the application. If you forgot to mount the vault-secrets volume in the app container, or mounted it elsewhere, the app wouldn’t see the file.
The next logical step is to explore how to use templates within the sync block to dynamically generate configuration files for your application, rather than just dumping raw secret values.