journalctl can make you a log-querying ninja, but most people are just scratching the surface. The most surprising thing is how much less data you often need to look at to find what you’re looking for, thanks to its structured querying.
Let’s see it in action. Imagine you’re debugging a service called my-app.service. You want to see its logs from the last hour, but only the errors.
journalctl -u my-app.service -p err -S -1h
This command does a few things:
-u my-app.service: Filters logs specifically for themy-app.serviceunit.-p err: Filters for messages with a priority oferroror higher (critical, alert, emergency).-S -1h: Sets the "since" time to one hour ago.
The output will look something like this, showing only the relevant error messages:
Sep 14 10:30:01 myhost my-app.service[12345]: ERROR: Database connection failed.
Sep 14 10:45:15 myhost my-app.service[12345]: ERROR: User ID 987 invalid.
This is powerful because systemd (which journald is part of) doesn’t just treat logs as plain text. It parses them, assigns priorities, and associates them with specific services, boot IDs, and more. journalctl is the interface to all that structured data.
Here’s how to build a more complex mental model.
The Problem It Solves: Traditional syslog can be a mess. Logs are often unstructured text files, making it hard to filter by service, priority, time ranges, or specific events without complex grep patterns. journald and journalctl bring structure and powerful querying to the table.
How It Works Internally: journald runs as a system daemon, collecting log messages from various sources: kernel, system services (via syslog or direct API), and stdin of services that systemd starts. It stores these logs in binary, indexed files (typically in /var/log/journal/). These indexes allow journalctl to perform very fast lookups based on various metadata fields.
The Exact Levers You Control:
- Units (
-u): Filter by service name (nginx.service,sshd.service), target (multi-user.target), or even kernel boot messages (-k). - Priorities (
-p): Filter by severity. The levels, from most to least severe, are:emerg(0),alert(1),crit(2),err(3),warning(4),notice(5),info(6),debug(7). You can specify a single level (e.g.,-p err) or a range (e.g.,-p warning..debug). - Time (
-S,-U):-S(Since): Specify a start time. Can be absolute (-S 2023-09-14 10:00:00) or relative (-S -1h,-S yesterday).-U(Until): Specify an end time. Same formats as-S.- Combining them (
-S yesterday -U today) gives you a precise window.
- Boot ID (
-b): Filter logs from a specific boot.journalctl -bshows logs from the current boot.journalctl -b -1shows logs from the previous boot. - Fields (
-F): Query for specific fields and their values. For example,journalctl -F MESSAGEwill show all messages.journalctl -F _PID=12345will show all logs from process ID 12345. - Output Formatting (
-o): Control how logs are displayed.short(default): Compact format.verbose: Shows all fields.json: Outputs logs as JSON objects, great for programmatic parsing.cat: Outputs only the message content, no metadata.
For example, to see all debug level messages from sshd in JSON format from the last 15 minutes, you’d use:
journalctl -u sshd.service -p debug -S -15m -o json
The one thing most people don’t realize is that journalctl can query based on any field that journald captures, not just the common ones. This includes things like _COMM (command name), _PID, _UID, _GID, _SELINUX_CONTEXT, and even custom fields you might add. You can discover available fields by looking at the output of journalctl -v or journalctl -o verbose. For instance, if your application logs a specific REQUEST_ID, you can filter all related logs with journalctl -F REQUEST_ID=abc123xyz.
The next step is understanding how to configure journald itself to manage disk space and retention policies.