strace lets you see what a process is doing by showing its system calls and signals.
Let’s see strace in action. Imagine you have a simple Python script that tries to read a file:
# read_file.py
try:
with open("my_data.txt", "r") as f:
content = f.read()
print(content)
except FileNotFoundError:
print("File not found!")
If my_data.txt doesn’t exist, and you run python read_file.py, you’ll see "File not found!". But what system call actually reported that the file was missing? That’s where strace comes in.
To see strace in action, first create the file:
echo "Hello, strace!" > my_data.txt
Now, run the Python script with strace:
strace python read_file.py
You’ll see a lot of output. Look for lines that involve file operations. You’ll see something like this (truncated for clarity):
execve("/usr/bin/python3", ["python", "read_file.py"], 0x7ffd8573a9e0 /* 69 vars */) = 0
...
openat(AT_FDCWD, "my_data.txt", O_RDONLY) = 3
read(3, "Hello, strace!\n", 4096) = 15
write(1, "Hello, strace!\n", 15) = 15
close(3) = 0
...
exit_group(0) = ?
+++ exited with 0 +++
This output shows the sequence of system calls. execve starts the Python interpreter. openat attempts to open my_data.txt. The 3 is the file descriptor returned by openat. read then reads from file descriptor 3. write sends the content to standard output (file descriptor 1). close cleans up.
Now, let’s remove the file and run strace again:
rm my_data.txt
strace python read_file.py
The output will change:
execve("/usr/bin/python3", ["python", "read_file.py"], 0x7ffdd74549e0 /* 69 vars */) = 0
...
openat(AT_FDCWD, "my_data.txt", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "File not found!\n", 16) = 16
...
exit_group(0) = ?
+++ exited with 0 +++
Notice the openat call now returns -1 and includes ENOENT (No such file or directory). This is the crucial piece of information strace provides: the kernel’s direct response to the program’s request. The Python script then catches this error and prints "File not found!" to standard error (file descriptor 2).
strace is your window into how a program interacts with the operating system. It shows every request a program makes to the kernel (system calls) and every signal it receives. This is invaluable for understanding why a program behaves the way it does, especially when it fails.
The Core Problem strace Solves
At its heart, strace solves the problem of "What is my program actually doing when it hangs, crashes, or misbehaves?" Most programming languages abstract away direct OS interactions. When something goes wrong at that low level, your high-level debugger or print statements might not show you the root cause. strace bridges that gap by showing the raw communication between your process and the kernel.
How strace Works Internally
strace uses the ptrace system call, a powerful but complex mechanism provided by the Linux kernel. When you run strace -p <pid>, it attaches to the target process. The kernel then stops the target process before each system call and after each system call. strace (running as a separate process) is notified by the kernel. It then inspects the process’s state (registers, memory) to decode the system call arguments and return values, prints them, and then tells the kernel to resume the target process. This "stop-and-inspect" cycle is repeated for every system call.
Controlling the Output
You can tailor strace’s output significantly.
Filtering System Calls:
If you only care about file operations, you can use -e trace=file.
strace -e trace=file python read_file.py
This would show only openat, read, write, close, etc.
Following Child Processes: If your program forks or execs other processes, you’ll want to follow them.
strace -f python my_parent_script.py
This will trace all descendant processes, which is essential for debugging multi-process applications.
Showing Timestamps: Understanding timing can be critical for performance issues.
strace -t python read_file.py
The -t flag adds the time of day. -tt adds microseconds, and -ttt adds microseconds since the epoch.
Showing Network Activity: For network-heavy applications, filtering for network calls is key.
strace -e trace=network python my_network_app.py
This will show calls like socket, connect, sendto, recvfrom, etc.
Decoding String Arguments:
By default, strace might truncate long strings. -s controls the string size.
strace -s 1024 python read_file.py
This ensures you see up to 1024 characters of any string argument, like file paths or buffer contents.
Attaching to a Running Process:
This is incredibly useful for debugging live applications. Find the process ID (PID) using ps aux | grep my_process or pgrep my_process.
# Assuming your process has PID 12345
strace -p 12345
You can also attach and then detach cleanly using Ctrl+C in strace without killing the target process.
Saving Output to a File:
Redirecting strace output is standard practice for later analysis.
strace -o strace.log python read_file.py
The -o option writes all strace output to strace.log. This is much cleaner than piping.
The One Thing Most People Don’t Know
When a system call fails, strace shows the error code (e.g., ENOENT). However, it also shows the errno value in parentheses. This errno value is a standard C library convention where specific integer codes map to specific error conditions. strace decodes these integer codes into human-readable strings like ENOENT or EACCES, which is incredibly helpful. But understanding that these are just standard POSIX error codes means you can look them up in man errno to get a definitive explanation of what went wrong at the kernel level, even beyond what strace’s decoding might imply.
The next thing you’ll want to master is using strace in conjunction with other tools like lsof to get a complete picture of resource usage.