The strace output format is a detailed, line-by-line log of system calls and signals a process makes or receives, and understanding it is key to debugging.
Let’s see strace in action. Imagine we’re tracing a simple cat command on a non-existent file:
strace cat /tmp/nonexistent_file
Here’s a snippet of what you might see:
execve("/bin/cat", ["cat", "/tmp/nonexistent_file"], 0x7ffc5c4816c0 /* 59 vars */) = 0
brk(NULL) = 0x5617367e4000
access("/etc/ld.so.preload", R_OK) = -1 ENOENT (No such file or directory)
openat(AT_FDCWD, "/etc/ld.so.cache", O_RDONLY|O_CLOEXEC) = 3
fstat(3, {st_mode=S_IFREG|0644, st_size=138648, ...}) = 0
mmap(NULL, 138648, PROT_READ, MAP_PRIVATE, 3, 0) = 0x7f929466b000
close(3) = 0
openat(AT_FDDIR, "/lib", O_RDONLY|O_NONBLOCK|O_CLOEXEC|O_DIRECTORY) = 3
...
openat(AT_FDCWD, "/tmp/nonexistent_file", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "cat: /tmp/nonexistent_file: No such file or directory\n", 61) = 61
exit_group(1) = ?
+++ exited with 1 +++
The problem strace solves is making the "black box" of the operating system’s interaction with a program visible. When a program needs to read a file, allocate memory, or send data over the network, it doesn’t do it directly; it asks the kernel to do it via a system call. strace intercepts these requests, logs them, and shows the results.
Each line in the strace output generally follows this structure: syscall_name(argument1, argument2, ...) = return_value.
syscall_name: This is the name of the system call being made. In the example,execve,brk,access,openat,write, andexit_groupare all system calls.- Arguments: These are the parameters passed to the system call. Their types and meanings depend entirely on the specific system call.
openatfor instance, takes a directory file descriptor (AT_FDCWDmeans the current working directory), a path ("/tmp/nonexistent_file"), and flags (O_RDONLY). return_value: This is what the kernel returned to the program.- A non-negative integer usually represents a successful result, often a file descriptor or the number of bytes read/written.
- A negative value, most commonly
-1, indicates an error.
- Error Information: When an error occurs (return value is
-1),stracehelpfully appends the error code and its human-readable description.ENOENT (No such file or directory)is a classic example, telling us the file or directory specified in theopenatcall simply doesn’t exist.
The execve call at the beginning is the first step in running a new program. It replaces the current process image with a new one. The brk system call is used for dynamic memory allocation; the program asks the kernel to extend its data segment. access checks file permissions. openat is a modern, more flexible way to open files, especially useful when dealing with path components relative to a directory file descriptor. The write call shows the program writing its error message to standard error (file descriptor 2). Finally, exit_group signals that the process and all its threads are terminating.
The +++ exited with 1 +++ line at the end indicates the process terminated with an exit status of 1, which is conventionally used for general errors.
The AT_FDCWD constant used in openat is a special value that means the path is interpreted relative to the current working directory of the calling process. Other flags like O_RDONLY (open for reading), O_CLOEXEC (close on exec), and O_DIRECTORY (fail if not a directory) modify the behavior of the openat call.
When parsing strace output, pay close attention to calls that return -1 and the associated error codes. ENOENT, EACCES (permission denied), EBUSY (device or resource busy), ENOMEM (out of memory), and EPIPE (broken pipe) are some of the most frequent culprits for application misbehavior. The arguments to these failing calls are crucial for pinpointing why the system call failed.
Understanding the various flags used with system calls like openat is also vital. For instance, O_CREAT would imply an attempt to create a file, which might fail if permissions are insufficient or the file already exists and O_EXCL wasn’t specified.
The most surprising thing about strace output is how much detail it reveals about even seemingly simple operations, often exposing subtle interactions between libraries, the dynamic linker, and the kernel that are rarely considered.
The next concept you’ll likely grapple with is how to filter strace output effectively to focus on specific system calls or processes, as tracing an entire application can generate an overwhelming amount of data.