strace is often seen as a last resort, a blunt instrument for debugging. The surprising truth is that it’s a precise microscope, revealing the kernel’s perspective on your process’s every interaction.
Let’s watch strace in action, tracing a simple ls command.
strace ls /tmp
The output scrolls by rapidly. You see system calls like execve (to start ls), openat (to look for directories and files), read (to get directory contents), and write (to display output).
execve("ls", ["ls", "/tmp"], 0x7ffc8d87d0c0 /* 55 vars */) = 0
brk(NULL) = 0x55a99c1d0000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=141459, ...}) = 0
mmap(NULL, 141459, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f32c3248000
close(3) = 0
openat(AT_FDCWD, "/lib/x86_64-linux-gnu/libc.so.6", O_RDONLY|O_CLOEXEC) = 3
read(3, "\177ELF\2\1\1\3\0\0\0\0\0\0\0\2\0\3\0\1\0\0\0\340\226\1\0\0\0\0\0"..., 832) = 832
fstat(3, {st_mode=S_IFREG|0755, st_size=2360280, ...}) = 0
mmap(NULL, 2363392, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f32c2e04000
mprotect(0x7f32c2e04000, 2097152, PROT_NONE) = 0
mmap(0x7f32c2e04000, 1851392, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e0000) = 0x7f32c2e04000
mmap(0x7f32c3000000, 348160, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x3c0000) = 0x7f32c3000000
mmap(0x7f32c3056000, 65536, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f32c3056000
close(3) = 0
openat(AT_FDCWD, "/etc/gai.conf", O_RDONLY) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/tmp", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
getdents64(3, /* 2 entries */, 32768) = 80
read(3, "\311\1\0\0\0\0\0\0\342\1\0\0\0\0\0\0", 32768) = 80
getdents64(3, /* 0 entries */, 32768) = 0
close(3) = 0
write(1, "file1\nfile2\n", 12) = 12
exit_group(0) = ?
+++ exited with 0 +++
The core problem strace solves is bridging the user-space/kernel-space divide. Your application lives in user-space, managing its own memory and execution. When it needs to do something fundamental – like read a file, create a network connection, or allocate memory – it can’t do it directly. It must ask the kernel to perform the action on its behalf. These requests are system calls. strace intercepts and displays these system calls, showing you exactly what your program is asking the kernel to do and what the kernel’s response is.
The mental model for strace is a conversation between your process and the kernel. strace is the eavesdropper, transcribing the entire dialogue. You see the request (the system call name and its arguments) and the response (the return value, which is often an error code).
When debugging, you’re looking for patterns in this conversation. For example, if your program is slow, you might see it making many identical system calls repeatedly, or it might be stuck in a loop of read and write calls that aren’t advancing its state. If it’s failing, you’ll see system calls returning error codes like ENOENT (No such file or directory), EACCES (Permission denied), or ENOMEM (Out of memory).
To attach strace to a running process, you use the -p flag:
strace -p 12345
Where 12345 is the PID of the process you want to trace. To follow new threads spawned by the traced process, use -f:
strace -p 12345 -f
This is crucial for multi-threaded applications where the issue might be in a child thread.
The most common pitfall is not understanding the return codes. A return value of -1 is almost always an error. The second value printed after it, e.g., ENOENT, is the specific error code. You can look these up with man errno. For instance, man 2 ENOENT will tell you what ENOENT means.
If your process is failing to start, you can run it directly under strace:
strace your_command_here
The key to effective strace usage is filtering. Raw strace output can be overwhelming. Use -e trace=syscall_name to focus on specific system calls. For example, to see only file-related operations:
strace -e trace=open,read,write,stat,close ls /tmp
Or to exclude certain calls that are too noisy:
strace -e trace=!futex,clock_gettime ls /tmp
The -s strsize option is vital for seeing the full content of strings passed to system calls, like file paths or network buffers. The default is usually 32 characters, which is often too short.
strace -s 1024 your_command_here
This setting is particularly important when dealing with network protocols or complex configuration files where long strings are common.
A common scenario for performance issues is excessive read or write calls to disk or network sockets. By using strace -c, you get a summary of system call counts and time spent, which can quickly highlight the most frequent or time-consuming operations.
strace -c your_command_here
This will produce a table showing the percentage of time spent in each system call, the number of calls, and the average time per call. This is invaluable for identifying bottlenecks.
When diagnosing network issues, paying close attention to connect, sendto, recvfrom, and poll/select calls is key. An ECONNREFUSED on connect means the server isn’t listening. Repeated EAGAIN or EWOULDBLOCK on recvfrom might indicate a busy server or a client not reading fast enough.
The mechanism by which strace works is through the PTRACE_TRACEME and PTRACE_SYSCALL ptrace requests. When you run a command with strace, strace itself becomes the parent. It calls ptrace(PTRACE_TRACEME, 0, NULL, NULL) for its child process (the command you’re tracing). This tells the kernel that the child wants to be traced by its parent. Then, strace uses ptrace(PTRACE_SYSCALL, pid, NULL, NULL) to tell the kernel to stop the child process just before it enters a system call and just after it returns from one. strace then reads the system call information and resumes the child.
The next thing you’ll likely encounter is understanding how to correlate strace output with kernel messages in dmesg.