Debug systemd Unit Files: journalctl and systemctl status (2026)

systemctl status is lying to you, or at least, it’s only telling you half the story about why your systemd service isn’t starting.

Let’s say you’ve just deployed a new service, my-awesome-app.service, and you try to start it:

sudo systemctl start my-awesome-app.service

And then you check its status:

sudo systemctl status my-awesome-app.service

You see something like this:

● my-awesome-app.service - My Awesome Application
     Loaded: loaded (/etc/systemd/system/my-awesome-app.service; enabled; vendor preset: enabled)
     Active: failed (result=exit-code) since Mon 2023-10-27 10:00:00 UTC; 5s ago
    Process: 12345 ExecStart=/usr/local/bin/my-awesome-app (code=exited, status=1/FAILURE)
   Main PID: 12345 (code=exited, status=1/FAILURE)

Oct 27 10:00:00 your-server systemd[1]: Starting My Awesome Application...
Oct 27 10:00:00 your-server my-awesome-app[12345]: Error: Configuration file not found at /etc/my-awesome-app/config.yml
Oct 27 10:00:00 your-server systemd[1]: my-awesome-app.service: Main process exited, code=exited, status=1/FAILURE
Oct 27 10:00:00 your-server systemd[1]: my-awesome-app.service: Failed with result 'exit-code'.
Oct 27 10:00:00 your-server systemd[1]: Failed to start My Awesome Application.

The systemctl status output shows failed (result=exit-code) and a cryptic status=1/FAILURE. This tells you that it failed, but not why in a way you can act on immediately. The real story is in the systemd journal.

The journal is systemd’s centralized logging system. Every service, every kernel message, every boot event – it all goes into journald. systemctl status pulls a snippet from this journal, but you need to dive deeper.

To see the actual logs for your service, you use journalctl. The most common way to filter for your specific service is with the -u flag:

sudo journalctl -u my-awesome-app.service

This will show you all the logs related to my-awesome-app.service since the journald service started. You’ll see the same error message from systemctl status, but potentially with more context from earlier in the boot process or from previous attempts to start the service.

Here’s the real power: you can follow the logs in real-time as you try to start or restart your service. This is invaluable for debugging.

sudo journalctl -f -u my-awesome-app.service

Now, while that command is running in one terminal, go to another and try starting your service again:

sudo systemctl start my-awesome-app.service

You’ll see the logs stream into your first terminal, giving you immediate feedback.

When systemctl status shows failed, it means the main process of your service exited with a non-zero status code. This is the universal signal for "something went wrong." For my-awesome-app.service, the journal log clearly states Error: Configuration file not found at /etc/my-awesome-app/config.yml.

Here are the most common reasons a service fails to start, and how to diagnose them with journalctl:

1. Missing or Incorrect Configuration Files: This is what we saw above. The application tried to read a config file that doesn’t exist or is malformed.

   • Diagnosis: sudo journalctl -u my-awesome-app.service and look for lines mentioning "config," "file not found," "permission denied," or specific config file paths.
   • Fix: Ensure the file exists at the specified path (/etc/my-awesome-app/config.yml in our example). Check its ownership and permissions. For example, if the app runs as the appuser user, that user needs read permission on the file: sudo chown appuser:appgroup /etc/my-awesome-app/config.yml and sudo chmod 644 /etc/my-awesome-app/config.yml.
   • Why it works: The application can now read the configuration it needs to initialize correctly.

2. Incorrect Permissions on Executables or Directories: The user your service runs as might not have permission to execute its own binary or access necessary directories.

   • Diagnosis: sudo journalctl -u my-awesome-app.service and look for "permission denied" errors when trying to execute a file or access a directory.
   • Fix: Ensure the executable has execute permissions for the user it runs as (often specified with User= in the unit file). For example, if my-awesome-app is in /usr/local/bin/ and runs as appuser: sudo chown appuser:appgroup /usr/local/bin/my-awesome-app and sudo chmod 755 /usr/local/bin/my-awesome-app. Also check permissions on any directories the app needs to write to or read from.
   • Why it works: The operating system allows the process to execute its code and interact with the filesystem.

3. Network Port Conflicts: The service tries to bind to a port that’s already in use by another process.

   • Diagnosis: sudo journalctl -u my-awesome-app.service will likely show an error like "Address already in use" or similar, often related to bind() system calls.
   • Fix: Identify the process using the port with sudo ss -tulnp | grep :8080 (replace 8080 with your service’s port). Stop the conflicting process, or change your service’s port in its configuration or unit file. If you need to change it in the unit file, e.g., from ExecStart=/usr/local/bin/my-awesome-app to ExecStart=/usr/local/bin/my-awesome-app --port 8081, then run sudo systemctl daemon-reload and sudo systemctl restart my-awesome-app.service.
   • Why it works: Your service can successfully bind to an available network port.

4. Missing Dependencies (Libraries or Other Services): The application depends on a shared library that isn’t installed, or another systemd service it needs to communicate with isn’t running or available.

   • Diagnosis: sudo journalctl -u my-awesome-app.service might show errors like "error while loading shared libraries" or specific messages indicating a failure to connect to another service.
   • Fix: For missing libraries, install them using your package manager (e.g., sudo apt-get install libssl-dev or sudo yum install openssl-devel). For missing services, ensure the dependency is listed in your unit file’s Requires= or Wants= directives and that the dependency service is running (sudo systemctl status other-service.service).
   • Why it works: The application finds its required libraries at runtime, or its dependencies are met before it starts.

5. Incorrect WorkingDirectory=: The WorkingDirectory= directive in the unit file points to a non-existent or inaccessible directory.

   • Diagnosis: sudo journalctl -u my-awesome-app.service will show errors related to file operations failing, often mentioning the WorkingDirectory path.
   • Fix: Correct the WorkingDirectory= path in your /etc/systemd/system/my-awesome-app.service file to a valid directory (e.g., WorkingDirectory=/opt/my-awesome-app/data). After editing, run sudo systemctl daemon-reload and sudo systemctl restart my-awesome-app.service.
   • Why it works: The service’s process starts in a valid directory from which it can perform its operations.

6. Syntax Errors in the Unit File: While less common for simple failures, a typo in ExecStart, User=, Group=, or other directives can cause systemd itself to reject the unit or start it incorrectly.

   • Diagnosis: sudo systemctl status my-awesome-app.service might show a slightly different error, or sudo journalctl -u my-awesome-app.service might contain messages from systemd itself indicating an issue parsing the unit file. Running sudo systemctl daemon-reload and observing its output is also a good check.
   • Fix: Carefully review your /etc/systemd/system/my-awesome-app.service file for typos, incorrect directive names, or missing required fields.
   • Why it works: Systemd correctly interprets the instructions for starting and managing your service.

Remember, journalctl is your primary tool. systemctl status is just the summary.

After fixing these, your next problem might be an OOM (Out Of Memory) killer situation if your application consumes too much RAM.