You’re trying to get a shell inside a Kubernetes pod, probably to poke around and see what’s going on, and kubectl exec is your go-to tool. But sometimes, it feels like you’re just staring at a blinking cursor, or worse, the command just hangs, and you’re left wondering why you can’t even get a simple shell.
Common Causes and Fixes for kubectl exec Failures
The core of the issue is that kubectl exec relies on the Kubernetes API server talking to the Kubelet on the node where your pod is running. If that communication path is broken, or if the pod itself isn’t set up to accept commands, exec will fail.
-
Pod Not Running or Not Ready: The most straightforward reason is that the pod simply isn’t in a state where it can accept connections.
- Diagnosis:
kubectl get pods -n <namespace> -o wide(Look forStatuslikePending,ContainerCreating,CrashLoopBackOff, orError. Check theREADYcolumn - it should be1/1or similar.) - Fix: If the pod is
Pending, checkkubectl describe pod <pod-name> -n <namespace>for reasons like insufficient resources or image pull errors. If it’sCrashLoopBackOff, checkkubectl logs <pod-name> -n <namespace> --previousto see why it’s crashing. ForContainerCreating, checkkubectl describe podfor image pull issues or volume mounting problems. Once the pod isRunningandREADY,execshould work. - Why it works:
kubectl execneeds a running container process to attach to. If the pod isn’t healthy, there’s no process toexecinto.
- Diagnosis:
-
Incorrect Container Name: If your pod has multiple containers, you need to specify which one you want to
execinto. Forgetting this is a common oversight.- Diagnosis:
kubectl describe pod <pod-name> -n <namespace>(Look for theContainers:section, which lists all container names). - Fix:
kubectl exec -n <namespace> -it <pod-name> -c <container-name> -- /bin/bash(Replace<container-name>with the actual name fromdescribe). - Why it works: Explicitly telling
kubectlwhich container to connect to resolves ambiguity and targets the correct process.
- Diagnosis:
-
execDisabled by Security Policies (PodSecurityPolicies/PodSecurityAdmission): Your cluster might have security policies preventing arbitrary command execution within pods.- Diagnosis: This is harder to diagnose directly from
kubectl execoutput alone. The error message might be generic like "command exited with 1" or "unable to upgrade connection." Check your cluster’s security configurations. For PSPs (now deprecated),kubectl auth can-i --as system:serviceaccount:default:default --as-group system:serviceaccounts --as-group system:authenticated create podsmight give clues, but it’s indirect. For PSA, check thePodSecurityadmission controller logs on your control plane nodes. - Fix: This depends heavily on your cluster’s security posture. You might need to:
- Adjust
PodSecurityPolicydefinitions (if still in use) to allowexecin certain namespaces or for specific service accounts. - Modify
PodSecurityAdmissionconfigurations (the modern approach) to exempt certain namespaces or relaxrestrictedprofiles. - For debug containers (discussed later), ensure the security context allows them.
- Adjust
- Why it works: Security policies are designed to prevent unauthorized actions, and
execinto a running container is considered a potentially risky operation if not managed.
- Diagnosis: This is harder to diagnose directly from
-
Network Connectivity Issues (API Server <-> Kubelet): The Kubelet on the node needs to be reachable by the API server for
execto work. Firewalls or network misconfigurations can block this.- Diagnosis:
kubectl get pod <pod-name> -n <namespace> -o wide(Note theNODEthe pod is running on). Then, try topingorcurlthe Kubelet’s IP address from a machine that can reach the API server (or vice-versa, if you have node access). The Kubelet typically runs on port10250(read-only) or10255(read-write, used byexec). Check firewall rules between your API server and the worker nodes. - Fix: Ensure that the firewall rules allow traffic on the Kubelet’s
execport (usually10250or10255) from the API server’s IP range to the worker nodes. If using a cloud provider, check security groups. - Why it works:
kubectl execis a proxied connection. The API server forwards your request to the Kubelet on the node, which then executes the command in the container. This chain needs network connectivity.
- Diagnosis:
-
Missing Shell or
execBinary in Container: The container image might be extremely minimal and not include a shell like/bin/bashor/bin/sh, or thekubectlbinary might not be able to find it.- Diagnosis: Try
kubectl exec -n <namespace> <pod-name> -- ls /bin/. If you don’t seebashorsh, that’s your problem. You can also trykubectl exec -n <namespace> <pod-name> -- /busybox/shif you suspect a different path. - Fix: Rebuild your container image to include a shell, or use a base image that has one. If you absolutely must
execinto a minimal image, you might need to specify a different shell if one exists (e.g.,kubectl exec -n <namespace> <pod-name> -- /bin/sh). - Why it works:
kubectl execrequires a shell process to be available within the container’s filesystem to launch commands.
- Diagnosis: Try
-
Kubelet Not Responding or Stuck: The Kubelet process on the worker node might be unhealthy, restarting, or unresponsive.
- Diagnosis: Log into the worker node where the pod is running. Check the status of the Kubelet service:
sudo systemctl status kubelet. Look for recent errors in its logs:sudo journalctl -u kubelet -f. - Fix: Restart the Kubelet service:
sudo systemctl restart kubelet. If it continues to fail, investigate the Kubelet logs for more specific issues (e.g., CNI plugin problems, certificate issues, disk pressure). - Why it works: The Kubelet is the agent on each node responsible for managing pods and communicating with the API server. If it’s down,
exec(and most other pod operations) will fail.
- Diagnosis: Log into the worker node where the pod is running. Check the status of the Kubelet service:
Debug Containers: A More Robust Approach
If you frequently need to exec into pods, especially minimal ones, consider using ephemeral containers (also known as debug containers). These are temporary containers you can launch into a running pod, providing a separate environment with debugging tools.
- How to use:
kubectl debug -it -n <namespace> <pod-name> --image=<debug-image> --target=<container-name>For example:kubectl debug -it -n default my-app-pod --image=ubuntu:latest --target=my-app-container
This command launches a new container from ubuntu:latest inside the my-app-pod namespace, running alongside my-app-container. You’ll get a shell into the ubuntu container, and from there, you can often access the filesystem of the target container (if volumes are shared) or use network tools to inspect the target container’s environment.
The next error you’ll likely hit after fixing kubectl exec is related to resource limits within the pod, causing your application to be OOMKilled.