You’re trying to get a shell inside a Kubernetes pod, probably to poke around and see what’s going on, and kubectl exec is your go-to tool. But sometimes, it feels like you’re just staring at a blinking cursor, or worse, the command just hangs, and you’re left wondering why you can’t even get a simple shell.

Common Causes and Fixes for kubectl exec Failures

The core of the issue is that kubectl exec relies on the Kubernetes API server talking to the Kubelet on the node where your pod is running. If that communication path is broken, or if the pod itself isn’t set up to accept commands, exec will fail.

  1. Pod Not Running or Not Ready: The most straightforward reason is that the pod simply isn’t in a state where it can accept connections.

    • Diagnosis: kubectl get pods -n <namespace> -o wide (Look for Status like Pending, ContainerCreating, CrashLoopBackOff, or Error. Check the READY column - it should be 1/1 or similar.)
    • Fix: If the pod is Pending, check kubectl describe pod <pod-name> -n <namespace> for reasons like insufficient resources or image pull errors. If it’s CrashLoopBackOff, check kubectl logs <pod-name> -n <namespace> --previous to see why it’s crashing. For ContainerCreating, check kubectl describe pod for image pull issues or volume mounting problems. Once the pod is Running and READY, exec should work.
    • Why it works: kubectl exec needs a running container process to attach to. If the pod isn’t healthy, there’s no process to exec into.
  2. Incorrect Container Name: If your pod has multiple containers, you need to specify which one you want to exec into. Forgetting this is a common oversight.

    • Diagnosis: kubectl describe pod <pod-name> -n <namespace> (Look for the Containers: section, which lists all container names).
    • Fix: kubectl exec -n <namespace> -it <pod-name> -c <container-name> -- /bin/bash (Replace <container-name> with the actual name from describe).
    • Why it works: Explicitly telling kubectl which container to connect to resolves ambiguity and targets the correct process.
  3. exec Disabled by Security Policies (PodSecurityPolicies/PodSecurityAdmission): Your cluster might have security policies preventing arbitrary command execution within pods.

    • Diagnosis: This is harder to diagnose directly from kubectl exec output alone. The error message might be generic like "command exited with 1" or "unable to upgrade connection." Check your cluster’s security configurations. For PSPs (now deprecated), kubectl auth can-i --as system:serviceaccount:default:default --as-group system:serviceaccounts --as-group system:authenticated create pods might give clues, but it’s indirect. For PSA, check the PodSecurity admission controller logs on your control plane nodes.
    • Fix: This depends heavily on your cluster’s security posture. You might need to:
      • Adjust PodSecurityPolicy definitions (if still in use) to allow exec in certain namespaces or for specific service accounts.
      • Modify PodSecurityAdmission configurations (the modern approach) to exempt certain namespaces or relax restricted profiles.
      • For debug containers (discussed later), ensure the security context allows them.
    • Why it works: Security policies are designed to prevent unauthorized actions, and exec into a running container is considered a potentially risky operation if not managed.
  4. Network Connectivity Issues (API Server <-> Kubelet): The Kubelet on the node needs to be reachable by the API server for exec to work. Firewalls or network misconfigurations can block this.

    • Diagnosis: kubectl get pod <pod-name> -n <namespace> -o wide (Note the NODE the pod is running on). Then, try to ping or curl the Kubelet’s IP address from a machine that can reach the API server (or vice-versa, if you have node access). The Kubelet typically runs on port 10250 (read-only) or 10255 (read-write, used by exec). Check firewall rules between your API server and the worker nodes.
    • Fix: Ensure that the firewall rules allow traffic on the Kubelet’s exec port (usually 10250 or 10255) from the API server’s IP range to the worker nodes. If using a cloud provider, check security groups.
    • Why it works: kubectl exec is a proxied connection. The API server forwards your request to the Kubelet on the node, which then executes the command in the container. This chain needs network connectivity.
  5. Missing Shell or exec Binary in Container: The container image might be extremely minimal and not include a shell like /bin/bash or /bin/sh, or the kubectl binary might not be able to find it.

    • Diagnosis: Try kubectl exec -n <namespace> <pod-name> -- ls /bin/. If you don’t see bash or sh, that’s your problem. You can also try kubectl exec -n <namespace> <pod-name> -- /busybox/sh if you suspect a different path.
    • Fix: Rebuild your container image to include a shell, or use a base image that has one. If you absolutely must exec into a minimal image, you might need to specify a different shell if one exists (e.g., kubectl exec -n <namespace> <pod-name> -- /bin/sh).
    • Why it works: kubectl exec requires a shell process to be available within the container’s filesystem to launch commands.
  6. Kubelet Not Responding or Stuck: The Kubelet process on the worker node might be unhealthy, restarting, or unresponsive.

    • Diagnosis: Log into the worker node where the pod is running. Check the status of the Kubelet service: sudo systemctl status kubelet. Look for recent errors in its logs: sudo journalctl -u kubelet -f.
    • Fix: Restart the Kubelet service: sudo systemctl restart kubelet. If it continues to fail, investigate the Kubelet logs for more specific issues (e.g., CNI plugin problems, certificate issues, disk pressure).
    • Why it works: The Kubelet is the agent on each node responsible for managing pods and communicating with the API server. If it’s down, exec (and most other pod operations) will fail.

Debug Containers: A More Robust Approach

If you frequently need to exec into pods, especially minimal ones, consider using ephemeral containers (also known as debug containers). These are temporary containers you can launch into a running pod, providing a separate environment with debugging tools.

  • How to use: kubectl debug -it -n <namespace> <pod-name> --image=<debug-image> --target=<container-name> For example: kubectl debug -it -n default my-app-pod --image=ubuntu:latest --target=my-app-container

This command launches a new container from ubuntu:latest inside the my-app-pod namespace, running alongside my-app-container. You’ll get a shell into the ubuntu container, and from there, you can often access the filesystem of the target container (if volumes are shared) or use network tools to inspect the target container’s environment.

The next error you’ll likely hit after fixing kubectl exec is related to resource limits within the pod, causing your application to be OOMKilled.

Want structured learning?

Take the full Ssh course →