strace is a powerful tool for debugging Go binaries, but it often shows way too much and can be overwhelming.
The Problem: Too Much Noise, Not Enough Signal
Go’s runtime and standard library make a lot of system calls. When you strace a simple Go program, you’ll be buried under thousands of read, write, poll, and futex calls that are just the Go scheduler and garbage collector doing their thing. It’s like trying to find a specific conversation in a stadium during a rock concert.
Seeing the Signal: Filtering strace Output
The key to using strace effectively with Go is filtering. You want to see your program’s interaction with the kernel, not the runtime’s internal mechanics.
1. Focus on Specific Syscalls
Most of the noise comes from low-level I/O and synchronization primitives like futex (used for mutexes and condition variables) and poll (used for waiting on I/O). You can exclude these:
strace -f -e trace=!futex,!poll,openat,read,write,close,connect,accept /path/to/your/go/binary
-f: Follow child processes. Crucial if your Go app spawns goroutines that become separate processes (less common) or if it uses libraries that fork.-e trace=!futex,!poll,...: This is the magic.!negates the syscall, so we’re tracing everything exceptfutexandpoll. We then explicitly include the syscalls we do care about:openat(file opening),read,write,close(file I/O),connect(network outgoing),accept(network incoming).
Why it works: By excluding the most frequent, noisy syscalls, you dramatically reduce the output volume, making it easier to spot the calls related to your application’s logic.
2. Trace Only Specific Processes
If your Go application spawns multiple processes (e.g., via os/exec), you might only care about the main process or a specific child.
# Trace only PID 12345
strace -p 12345 -f -e trace=!futex,!poll,openat,read,write,close,connect,accept
# Trace the main process and its children, but filter by a specific syscall pattern
strace -f -p $(pgrep -f your_go_binary) -e trace='write,read,openat'
-p PID: Attach to an already running process.$(pgrep -f your_go_binary): This command substitution finds the Process ID (PID) of your Go binary based on its name.
Why it works: Attaching to a specific PID or filtering by process name ensures you’re only observing the execution flow you’re interested in, cutting out unrelated system activity.
3. Filter by File Descriptors
Often, you’re interested in I/O to a specific file or network connection. You can filter strace output by file descriptor (FD).
# Trace all operations on FD 3 (e.g., stdout if redirected)
strace -f -e 'write=3,read=3' /path/to/your/go/binary
# Trace network connections to a specific remote IP/port
strace -f -e 'connect,accept' -s 1024 /path/to/your/go/binary | grep '-> 192.168.1.100:8080'
-s 1024: Increases the string length shown for arguments (like IP addresses and port numbers) to 1024 characters. The default is often too small.| grep ...: Standard Unix piping to filter the output ofstracefor specific patterns.
Why it works: File descriptors are kernel-level handles. By filtering on specific FDs or patterns in the syscall arguments, you isolate the I/O operations relevant to your debugging target.
4. Look for Specific Argument Patterns
Sometimes, the syscall itself is fine, but the arguments are wrong. For example, trying to open a non-existent file.
# Find all 'openat' calls that fail with ENOENT (No such file or directory)
strace -f -e trace=openat /path/to/your/go/binary 2>&1 | grep 'ENOENT'
# Find 'connect' calls attempting to reach a specific invalid host/port
strace -f -e trace=connect /path/to/your/go/binary 2>&1 | grep '-> 127.0.0.1:9999'
2>&1: Redirects standard error (wherestracewrites its output) to standard output, sogrepcan process it.
Why it works: This focuses your attention on syscalls that are returning errors or attempting operations with problematic parameters, pinpointing logical flaws in your Go code.
5. High-Level Go Runtime Events
While strace is fundamentally about syscalls, you can sometimes infer Go runtime behavior. For instance, a sudden burst of futex calls might indicate heavy contention on a mutex or channel. A lot of mmap and munmap calls could point to aggressive memory allocation/deallocation, possibly related to garbage collection cycles or large data structures.
# Observe mmap/munmap activity
strace -f -e trace=mmap,munmap /path/to/your/go/binary
Why it works: By observing patterns in these specific syscalls, you can get clues about the Go runtime’s internal state and resource management, even without understanding every single goroutine’s execution.
The Next Hurdle: Understanding Goroutine Scheduling
Once you’ve filtered out the noise and are looking at the relevant syscalls, the next challenge is correlating those syscalls back to specific goroutines. strace itself doesn’t directly show goroutine IDs. You’ll need to combine strace output with Go’s built-in profiling tools (pprof) or add custom logging to your Go code to map syscalls to the logical execution paths within your application.