strace in containers is surprisingly powerful for debugging complex interactions between your application and the kernel, especially when traditional logging or application-level debugging falls short.
Let’s see strace in action. Imagine you have a simple Dockerfile:
FROM ubuntu:latest
CMD ["/app/my_program"]
And a C program my_program.c:
#include <stdio.h>
#include <unistd.h>
#include <fcntl.h>
int main() {
int fd = open("/data/my_file.txt", O_RDWR | O_CREAT, 0666);
if (fd == -1) {
perror("Failed to open file");
return 1;
}
dprintf(fd, "Hello from strace demo!\n");
close(fd);
printf("File operation complete.\n");
return 0;
}
First, build the image and mount a volume:
docker build -t strace-demo .
mkdir my_data
docker run -it --rm -v $(pwd)/my_data:/data strace-demo
Now, to trace the syscalls:
docker run -it --rm -v $(pwd)/my_data:/data -e "STRACE_CMD=/app/my_program" --entrypoint strace strace-demo
This will show output like:
execve("/app/my_program", ["/app/my_program"], 0x7ffc4b0c67d0 /* 35 vars */) = 0
brk(NULL) = 0x55c741b3d000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=134744, ...}) = 0
mmap(NULL, 134744, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f7251000000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\2\0\3\0\1\0\0\0\320\34\2\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0644, st_size=2201136, ...}) = 0
mmap(NULL, 2103296, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f7250d9b000
mprotect(0x7f7250e01000, 1980416, PROT_NONE) = 0
mmap(0x7f7250e01000, 1643008, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x66000) = 0x7f7250e01000
mmap(0x7f7250fa1000, 315456, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1a6000) = 0x7f7250fa1000
mmap(0x7f7250ffb000, 65600, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f7250ffb000
close(3) = 0
openat(AT_FDCWD, "/app/my_program", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0755, st_size=8600, ...}) = 0
close(3) = 0
openat(AT_FDCWD, "/data/my_file.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3
fstat(3, {st_mode=S_IFREG|0666, st_size=0, ...}) = 0
write(3, "Hello from strace demo!\n", 26) = 26
close(3) = 0
write(1, "File operation complete.\n", 25) = 25
exit_group(0) = ?
+++ exited with 0 +++
The problem strace solves is observing the exact interface between your program and the Linux kernel. When your application behaves unexpectedly, and logs are insufficient or misleading, strace shows you what system calls are being made, their arguments, and their return values. This is crucial for understanding resource contention, permission issues, or unexpected kernel behavior.
The core of strace’s power lies in its ability to intercept and display system calls. Each line represents a single call made by your program to the operating system. For example, openat(AT_FDCWD, "/data/my_file.txt", O_WRONLY|O_CREAT|O_TRUNC, 0666) = 3 shows your program attempting to open a file named /data/my_file.txt for writing, creating it if it doesn’t exist, and truncating it if it does. The 0666 are the permissions, and 3 is the file descriptor returned by the kernel, which your program will use for subsequent operations on that file.
The mental model for strace is simple: your program asks the kernel to do something, strace watches that request and its response. In the context of Docker, this means you’re tracing the syscalls made by the containerized process, not directly by the host. This allows you to debug issues that are specific to the container’s environment, like incorrect file system mounts, missing capabilities, or network configuration problems.
To use strace within Docker, you typically run your container with strace as the entrypoint, or use docker exec to attach strace to an already running container. The -e strace flag is your best friend. You can also filter by syscall name with -e trace=openat,read,write for example, to reduce noise.
There’s a common misconception that strace is slow and only for extreme cases. While it does add overhead, for many debugging scenarios, the performance impact is negligible and far outweighed by the diagnostic information gained. The key is to understand that strace doesn’t just show what your program is doing, but how it’s interacting with the fundamental services provided by the operating system kernel. It’s the closest you can get to seeing the "source code" of the kernel’s interaction with your application.
When debugging file access issues within containers, pay close attention to openat calls and their return values. A return of -1 indicates an error, and the subsequent errno code (like ENOENT for "No such file or directory" or EACCES for "Permission denied") directly tells you why the operation failed. This is invaluable for diagnosing problems with volume mounts or file permissions that seem to be misconfigured.
The next hurdle you’ll often face after mastering strace is understanding how to interpret complex ioctl calls or network-related syscalls like sendmsg and recvmsg.