Articles

strace in Production: Safe Tracing Without Disruption (2026)

April 6, 2026 5 min read

The most surprising thing about strace is that it’s not inherently dangerous in production, provided you understand its impact and use it judiciously.

Let’s see strace in action. Imagine a web server process, PID 12345, that’s behaving sluggishly. We want to see what system calls it’s making.

strace -p 12345 -s 1024 -f -tt -T -o /tmp/webserver.strace

Here’s what’s happening:

  • -p 12345: Attach to the process with PID 12345. This is the core of tracing an existing process.
  • -s 1024: Set the string length to 1024 characters. By default, strace truncates strings, which can hide important data. This ensures we see more of what the process is actually passing to or receiving from the kernel.
  • -f: Follow forks. If the web server spawns child processes, strace will follow them too. Essential for multi-process applications.
  • -tt: Print microsecond-resolution timestamps for each system call. This lets us see the duration of each call and identify bottlenecks.
  • -T: Print the time spent in each system call. This is crucial for performance analysis.
  • -o /tmp/webserver.strace: Write the output to a file named webserver.strace in /tmp. This prevents flooding your terminal and provides a persistent record.

The output in /tmp/webserver.strace will look something like this:

14:30:01.123456 read(3, "GET / HTTP/1.1\r\nHost: example.com\r\nUser-Agent: curl/7.68.0\r\nAccept: */*\r\n\r\n", 1024) = 100 <0.000123>
14:30:01.123600 epoll_wait(4, [{EPOLLIN, {u32=12345678, u64=...}}], 512, 1000) = 1 <0.000087>
14:30:01.123750 accept4(5, {sa_family=AF_INET, sin_port=htons(54321), sin_addr=inet_addr("192.168.1.100")}, [16], 0) = 6 <0.000099>
14:30:01.123900 write(6, "HTTP/1.1 200 OK\r\nContent-Type: text/html\r\nContent-Length: 1024\r\n\r\n<html><body>Hello!</body></html>", 100) = 100 <0.000150>

This output tells us:

  • At 14:30:01.123456, the process read 100 bytes from file descriptor 3 (likely a network socket). The call took 0.000123 seconds.
  • It then waited for events on epoll descriptor 4, which returned one event after 0.000087 seconds.
  • It accepted a new connection on descriptor 5, resulting in a new descriptor 6 for the client. This took 0.000099 seconds.
  • Finally, it wrote 100 bytes to the new client socket (6), taking 0.000150 seconds.

The Mental Model: Kernel Interception

strace works by leveraging the ptrace system call. When strace attaches to a process, it tells the kernel, "Hey, whenever this process is about to make a system call, pause it and let me see what’s happening." The kernel then intercepts every system call the traced process makes. It stops the process, sends information about the system call (name, arguments, return value, duration) to strace, and then resumes the process.

This interception is the source of strace’s power and its perceived danger.

The Problem strace Solves

In production, you often face a "black box" application. It’s running, but you don’t know why it’s slow, why it’s erroring, or why it’s consuming resources. strace pulls back the curtain, showing you the exact interactions between your application and the operating system. This is invaluable for debugging:

  • Performance Bottlenecks: Identifying slow system calls (e.g., disk I/O, network operations, mutex waits).
  • Error Diagnosis: Seeing the specific system call that failed and its error code (e.g., ENOENT for "No such file or directory" when trying to open a config file).
  • Resource Leaks: Observing repeated, unexpected file descriptor openings or network connections.
  • Configuration Issues: Verifying if an application is reading configuration files from the expected location or if it’s trying to bind to an already-used port.

The Levers You Control

  • Which Process: Use -p PID to attach to a specific process, or strace -f -o output.log my_command to trace a new command from its inception.
  • What to Trace:
    • -e trace=syscall_name: Trace only specific system calls. For example, -e trace=open,read,write to focus on I/O.
    • -e trace=!read to trace everything except read.
    • -e signal=SIGNAL_NAME to trace specific signals.
  • Output Formatting:
    • -s STRINGLEN: Control string length.
    • -v: Verbose output (shows more detail for certain calls).
    • -x: Print non-ASCII strings in hex.
    • -f, -F: Follow forks/vforks.
  • Performance Impact:
    • -T: Show time spent in calls.
    • -r: Relative timestamps.
    • -c: Count system calls and summarize (less detail, lower overhead).

The Hidden Cost: Performance Overhead

The most critical thing to understand about strace is its performance impact. Every system call involves a context switch: the kernel stops the process, strace gets control, strace processes the info, the kernel resumes the process. If your application makes thousands of system calls per second (which many do), strace can easily slow it down by 2x, 10x, or even 100x. This is why you don’t run strace -f -o /dev/null /path/to/your/app as a permanent monitoring solution. You use it surgically.

The -c option significantly reduces this overhead by only counting calls and summarizing them at the end, rather than printing every single one. This is a good compromise if you need a general idea of system call activity without the full performance hit of detailed tracing.

The next logical step after identifying a problematic system call with strace is to understand how to interpret its return codes and errno values, often by consulting the man pages for that specific system call.

More Deep Dives in Strace

Want structured learning?

Take the full Strace course →