strace Debug Slow Application: Find Expensive Syscalls (2026)

strace is your best friend when an application is slow, and you suspect it’s getting bogged down making too many or too long syscalls. This isn’t about your application logic being slow; it’s about the operating system calls your application is making that are taking ages.

Let’s say you have a Python script, my_slow_app.py, that’s eating up CPU and not finishing.

# my_slow_app.py
import time
import os

def do_work():
    for _ in range(1000):
        # Simulate some I/O bound work
        with open("temp.txt", "a") as f:
            f.write("a" * 100)
        os.remove("temp.txt")
        time.sleep(0.001) # Small sleep to not peg CPU, but still be slow

if __name__ == "__main__":
    do_work()

You run it: python my_slow_app.py and it’s agonizingly slow.

The Diagnosis: What’s Actually Happening?

The core issue here is that your application is spending an excessive amount of time in system calls, specifically those related to file I/O (open, write, remove) and potentially others like read, close, stat, fstat, lseek, or even network-related calls (sendto, recvfrom, connect, accept). strace lets us peek under the hood and see exactly which syscalls are being made and how long they take.

Common Causes and How to Fix Them

1. Excessive File Operations (Open/Close Churn): Your application is opening and closing files repeatedly, often in a loop, instead of keeping them open or processing them in batches.

   • Diagnosis: Run strace -c python my_slow_app.py. This command counts the total number of each syscall and the percentage of time spent in each. Look for high counts on open, close, write, and unlink/remove.

     % time     seconds  calls errors syscall
     ------ ----------- ----------- --------- --------------------------------
      99.82    1.234567        1000           write
       0.10    0.000123        1000           open
       0.05    0.000067        1000           unlink
       0.03    0.000034        1000           close
     

   • Fix: Modify your application to minimize open/close cycles. For our example, we’d want to write all data to a single file, then close it once, and remove it once.

     # my_slow_app_fixed.py
     import time
     import os
     
     def do_work():
         with open("temp_batch.txt", "w") as f: # Open once
             for _ in range(1000):
                 f.write("a" * 100)
         os.remove("temp_batch.txt") # Remove once
         time.sleep(0.001)
     
     if __name__ == "__main__":
         do_work()
     

   • Why it works: Each open and close syscall involves kernel overhead. By reducing the number of these operations, you drastically cut down on that overhead, allowing the application to spend more time on actual data processing.

2. Large Data Transfers: Your application is reading or writing very large chunks of data in a single syscall, which can be inefficient if the underlying buffer management or network stack has to do a lot of work.

   • Diagnosis: Use strace -s 1024 -c python my_slow_app.py. The -s 1024 option increases the string length shown for syscall arguments, helping to identify large buffer sizes. Look at the write and read syscalls and their seconds and calls columns. If you see a few calls with very high seconds for write/read, it might indicate this.
   • Fix: If possible, break down large I/O operations into smaller, more manageable chunks. For example, instead of writing 1MB at once, write 64KB chunks. This is more about application design than a direct syscall fix, but strace points you to the problematic syscall.
   • Why it works: Smaller I/O operations can sometimes be handled more efficiently by the kernel and filesystem/network layers, avoiding potential buffer overflows or complex memory copying within the kernel.

3. Inefficient String/Memory Operations (e.g., mmap with small pages): While less common for simple scripts, applications doing heavy memory manipulation or memory-mapped file access might be hitting issues with how memory is managed by the kernel.

   • Diagnosis: Look for frequent mmap, munmap, brk, and mprotect syscalls. If these are high in count and take significant time, it might be an issue. strace -e trace=memory python my_slow_app.py can filter to show only memory-related syscalls.
   • Fix: This is highly application-specific. It might involve optimizing memory allocation patterns, using different data structures, or re-evaluating how memory-mapped files are accessed. For instance, if mmap is being called repeatedly on small, non-contiguous regions, it can be inefficient. Consolidating access to larger, contiguous regions can help.
   • Why it works: The kernel manages memory pages (typically 4KB). Frequent mmap calls for small, scattered regions can lead to increased overhead in tracking these mappings and managing page tables. Aligning operations to larger, page-aligned chunks improves efficiency.

4. Frequent fsync/fdatasync Calls: Applications that are overly cautious about data durability might call fsync after every write, which forces all buffered data for a file to be written to stable storage (disk).

   • Diagnosis: Run strace -c python my_slow_app.py. Look for a high count of fsync or fdatasync syscalls, especially if they are taking a noticeable amount of time.
   • Fix: Batch fsync operations. Instead of calling fsync after every small write, call it once after a group of writes or at logical checkpoints in your application’s execution.
   • Why it works: fsync is a synchronous operation that can be very slow because it involves disk I/O. Reducing its frequency allows the kernel to buffer writes more effectively and perform fewer, larger disk writes, which are generally more efficient.

5. Excessive stat/lstat/fstat Calls: Applications that check file metadata (permissions, size, timestamps) very frequently, especially in loops, can incur significant overhead.

   • Diagnosis: Use strace -c python my_slow_app.py and watch the counts for stat, lstat, and fstat. If these are called thousands or millions of times and consume a notable percentage of time, it’s a problem.
   • Fix: Cache file metadata. If you’ve just performed an operation on a file, you likely already have its metadata. Store it in a variable and reuse it instead of calling stat again.
   • Why it works: stat syscalls involve filesystem lookups. Repeatedly querying the filesystem for the same file’s metadata is redundant work that can be avoided by simply remembering the information once it’s retrieved.

6. Pathname Lookup Overhead (openat, statat, etc.): Modern Linux uses openat, statat, etc., which are more flexible but can still be slow if a deeply nested path is repeatedly looked up.

   • Diagnosis: Look for openat, statat in strace -c output. If you see these, especially with a large number of arguments indicating a complex path, and they are slow, this could be the culprit.
   • Fix: Similar to stat, cache directory file descriptors (fd) if possible. Instead of openat(AT_FDCWD, "/path/to/file", ...) repeatedly, use openat(dir_fd, "file", ...) where dir_fd is an open syscall on /path/to/ that’s kept open.
   • Why it works: AT_FDCWD requires the kernel to resolve the full path from the root of the filesystem every time. Using a file descriptor for a directory (dir_fd) allows the kernel to perform a faster, relative lookup within that directory’s inode.

After applying fixes for the most common causes, the next error you might encounter is related to resource exhaustion if your application’s performance has improved so much that it now consumes more CPU or memory than anticipated, leading to SIGKILL or out-of-memory errors.