strace and perf are both invaluable Linux tracing tools, but they operate at fundamentally different levels and are suited for distinct debugging scenarios.
Let’s see strace in action. Imagine a simple C program that tries to open a non-existent file:
#include <stdio.h>
#include <fcntl.h>
#include <errno.h>
#include <string.h>
int main() {
int fd = open("non_existent_file.txt", O_RDONLY);
if (fd == -1) {
fprintf(stderr, "Error opening file: %s\n", strerror(errno));
return 1;
}
printf("File opened successfully (this should not happen).\n");
close(fd);
return 0;
}
Now, let’s run strace on this program:
strace ./my_program
The output will be a verbose, line-by-line account of every system call the program makes:
execve(...) = 0
...
openat(AT_FDCWD, "non_existent_file.txt", O_RDONLY) = -1 ENOENT (No such file or directory)
write(2, "Error opening file: No such file"..., 42) = 42
exit_group(1) = ?
+++ exited with 1 +++
This output clearly shows the openat system call failing with ENOENT. strace essentially intercepts and logs interactions between userspace processes and the Linux kernel. It tells you what system calls a process is making, what arguments it’s passing, and what return values it’s getting. This is incredibly powerful for understanding how a program interacts with the operating system, especially when dealing with I/O, file operations, network sockets, or process management.
perf, on the other hand, is a performance analysis tool that leverages hardware performance counters and software tracepoints. It’s not about individual system calls but about understanding the performance characteristics of your code and system.
Consider a simple loop that does some computation:
#include <stdio.h>
long long sum_of_squares(int n) {
long long sum = 0;
for (int i = 0; i < n; ++i) {
sum += (long long)i * i;
}
return sum;
}
int main() {
long long result = sum_of_squares(1000000);
printf("Sum: %lld\n", result);
return 0;
}
To analyze its performance with perf, you might run:
perf record -e cycles:u ./my_compute_program
perf report
The perf record command will sample the cycles (CPU cycles) event in user mode (:u). perf report will then present a summary of where the program is spending its time, often showing that the sum_of_squares function is consuming a significant portion of the CPU cycles.
The core problem strace solves is understanding behavior and correctness at the system interface. When a program behaves unexpectedly – crashing, hanging, or returning incorrect results – strace can pinpoint the exact system call that’s misbehaving. It’s your go-to for debugging issues like:
- "Why can’t my program open this file?" (
open,statfailing) - "Why is my network request timing out?" (
connect,sendto,recvfromissues) - "Why is my process stuck in a loop?" (Observing repeated system calls)
- "What permissions is my program trying to access?" (Seeing
accesscalls)
perf is designed to answer questions about performance:
- "Why is my application slow?" (Identifying hot code paths)
- "What is causing high CPU utilization?" (Profiling CPU-bound tasks)
- "Are there cache misses or branch mispredictions impacting performance?" (Using hardware performance counters)
- "What is the latency of a specific operation?" (Using tracepoints for timing)
The mental model for strace is a transcript of a conversation between a process and the kernel. You’re reading the log of requests and responses. For perf, it’s more like a performance dashboard, showing you where the engine is revving the hardest, or where the bottlenecks are, often at a much lower level than individual system calls.
The common lever you pull with strace is filtering. You can limit the output to specific system calls (e.g., -e trace=open,read,write), specific processes, or even specific file descriptors. This is crucial because strace output can be overwhelmingly verbose.
With perf, your primary levers are the events you choose to sample (-e) and the context (:u for user, :k for kernel, :p for kernel pages) and the sampling frequency. You can drill down into specific functions, memory accesses, or even instruction pointers.
The most surprising thing about perf is how it can reveal performance issues that aren’t obvious from just looking at the code. It can highlight the impact of hardware architecture, memory access patterns, and even scheduler behavior in ways that static code analysis or strace simply cannot. For instance, you might find that a perfectly written algorithm is slow due to excessive cache misses, or that a few poorly placed memory accesses are causing pipeline stalls.
One thing most people don’t know is that perf can trace system calls too, but it does so by leveraging kernel tracepoints rather than by intercepting them like strace. This can be more efficient for high-volume tracing. You can use perf trace (which is distinct from perf record or perf report) for this:
perf trace -e 'syscalls:sys_enter_*' -p <PID>
This command will trace all system call entry events for a given PID, offering a perf-native alternative to strace that can sometimes be faster and integrate better with the broader perf ecosystem for analysis.
The next concept you’ll likely explore is how to combine strace and perf for a comprehensive view, using strace to understand what is happening at the system interface and perf to understand why it might be slow or inefficient.