You can filter strace’s output to focus on specific syscall groups like network, file, or IPC operations, which is crucial for debugging performance bottlenecks or understanding inter-process communication.
Let’s see strace in action. Imagine you’re trying to figure out why a web server is slow. A common culprit is excessive disk I/O. We can trace just the file-related system calls to pinpoint the problem.
Here’s how you’d trace a curl request, focusing only on file descriptor operations:
strace -e trace=file curl -s http://localhost:8080/large-file
This command will output only system calls related to file operations, such as open, read, write, close, stat, and lseek. You’ll see the sequence of file interactions that curl performs to fetch the file.
Now, let’s break down the mental model of how strace and its filtering work.
strace intercepts and records system calls made by a process. A system call is the interface between a user-space program and the operating system kernel. When a program needs to perform an action that requires kernel privileges, like reading a file or sending data over the network, it makes a system call. strace hooks into this process, showing you exactly what requests your program is making to the kernel.
The -e trace= option is the key to filtering. It allows you to specify which system calls you want to see. The kernel categorizes system calls into groups. The most common groups are:
file: Operations related to file descriptors and file system access. This includes calls likeopenat,read,write,close,lseek,stat,fstat,access,mkdirat,unlinkat,renameat,ioctl(when used on file descriptors).network: Operations related to network sockets and communication. This includes calls likesocket,bind,listen,accept,connect,sendto,recvfrom,sendmsg,recvmsg,getsockopt,setsockopt,shutdown,poll,select,epoll_wait.ipc: Operations related to Inter-Process Communication. This includes calls likepipe,pipe2,shmget,shmat,shmdt,semget,semop,msgget,msgsnd,msgrcv.process: Operations related to process management. This includes calls likefork,vfork,clone,execve,wait4,exit_group,kill,getpid,getppid.signal: Operations related to signal handling. This includes calls likert_sigaction,rt_sigprocmask,rt_sigreturn.desc: Operations related to descriptor management (a superset offile). This includesopenat,close,dup,dup2,fcntl,ioctl.memory: Operations related to memory management. This includesmmap,munmap,brk.
You can specify multiple groups by separating them with commas. For example, to trace both file and network operations:
strace -e trace=file,network curl -s http://localhost:8080/large-file
You can also specify individual system calls by name. For instance, to only see openat and read calls:
strace -e trace=openat,read curl -s http://localhost:8080/large-file
The power comes from combining these. If you suspect your application is having trouble with file descriptors, strace -e trace=desc is your go-to. If you’re debugging a distributed system and suspect network issues, strace -e trace=network will highlight all socket-related activities.
When you’re debugging a complex application, you might be tempted to strace -f (follow forks) and strace -e trace=all. This generates a massive amount of data, making it hard to see the forest for the trees. By carefully selecting the syscall groups relevant to your problem, you drastically reduce noise and focus on the precise interactions causing your issue. For example, if you’re seeing timeouts on a client connecting to a server, you’d want to see connect, sendto, and recvfrom calls, so strace -e trace=network is your starting point.
The strace output for a system call includes the call name, its arguments (decoded into human-readable forms where possible), and its return value. For example, an openat call might look like:
openat(AT_FDCWD, "/etc/passwd", O_RDONLY|O_CLOEXEC) = 3
Here, AT_FDCWD means the path is relative to the current working directory, /etc/passwd is the file being opened, O_RDONLY and O_CLOEXEC are flags, and 3 is the file descriptor returned by the kernel. If the call fails, the return value will be -1 and errno will be set, which strace also shows.
When you’re using strace and see a read or write call that takes a very long time, it’s often because the underlying operation is slow. For files, this could be a slow disk, a network filesystem (NFS, SMB), or even a virtualized disk. For network calls, it could be network latency, packet loss, or a slow remote endpoint.
The ioctl system call is a bit of a wildcard. It’s a general-purpose interface for device-specific operations. When you see ioctl in strace, especially when filtering for file, it’s often interacting with devices that behave like files (e.g., /dev/tty, /dev/input/mouse0, /dev/null). Its behavior is highly dependent on the file descriptor it’s called on, and strace will often show its arguments and return value, but the meaning of those can be obscure without understanding the underlying device driver.
The next step after mastering syscall group filtering is understanding how to combine strace with other debugging tools, like perf, for deeper performance analysis.