The systemd service failed because a file or directory specified in ExecStartPre command was not found by the systemd process. This often happens because systemd doesn’t inherit the user’s shell environment, including PATH and other variables, leading to commands or file paths that work interactively to fail when executed by systemd.
Here are the most common reasons and how to fix them:
1. Incorrect or Relative Path in ExecStartPre
Diagnosis:
The most straightforward cause is simply a typo or using a relative path that doesn’t resolve correctly within systemd’s execution context. systemd runs commands with a very minimal environment.
Command:
Manually try executing the exact command from your ExecStartPre in a minimal shell environment. You can simulate this by logging in as the user the service runs as (if applicable) and then running:
env -i /bin/bash --login -c 'YOUR_EXECSTARTPRE_COMMAND_HERE'
Replace YOUR_EXECSTARTPRE_COMMAND_HERE with the full command from your .service file.
Fix:
Use the absolute path to the command and any files/directories it needs.
For example, if your ExecStartPre is:
ExecStartPre=/usr/local/bin/my_script.sh --config /etc/my_app/config.yml
And my_script.sh is actually in /usr/local/bin/my_script.sh, but you wrote it as my_script.sh or assumed PATH would find it, this is the problem. Ensure the path is correct. If the script itself is the issue, make sure it’s executable (chmod +x /usr/local/bin/my_script.sh).
Why it works:
systemd executes commands directly without sourcing user profiles or assuming a typical shell PATH. Providing the full, absolute path ensures systemd knows exactly where to find the executable and any referenced files.
2. Missing Executable in PATH for systemd
Diagnosis:
The command specified in ExecStartPre exists, but it’s not in the PATH that systemd uses. systemd has a very restricted default PATH.
Command:
Check the PATH available to systemd by looking at the service’s status.
systemctl status your_service_name.service
Look for lines indicating the PATH or try to infer it by observing if simple commands like ls work but custom ones don’t. You can also try running env within an ExecStartPre to see what PATH is available:
ExecStartPre=env
This will print the environment variables to the service’s log, including the PATH.
Fix:
Specify the absolute path to the executable in ExecStartPre.
If your ExecStartPre is:
ExecStartPre=my_utility --setup
and my_utility is located at /opt/my_app/bin/my_utility, change it to:
ExecStartPre=/opt/my_app/bin/my_utility --setup
Alternatively, you can explicitly set the PATH for the service unit using Environment= or EnvironmentFile=.
[Service]
Environment="PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/opt/my_app/bin"
ExecStartPre=my_utility --setup
Why it works:
By providing the absolute path, you bypass the need for systemd to search the PATH. Setting Environment="PATH=..." explicitly tells systemd which directories to search, making your command discoverable.
3. File Permissions or Ownership Issues
Diagnosis:
The file or directory exists, but the user systemd is running the service as (specified by User= and Group= in the service file, or root by default) does not have permission to access it.
Command:
Use ls -l on the file or directory mentioned in ExecStartPre to check permissions.
ls -l /path/to/your/file_or_directory
Pay close attention to the owner, group, and the read/write/execute bits for the user, group, and others.
Fix:
Adjust permissions or ownership.
If the file is /var/myapp/data/init.lock and systemd is running as user myappuser, but init.lock is owned by root with permissions rw-r--r--, myappuser might not be able to create or modify it if ExecStartPre needs to do so.
You can change ownership:
chown myappuser:myappuser /var/myapp/data/init.lock
Or adjust permissions (use with caution):
chmod +w /var/myapp/data/init.lock
If ExecStartPre is a script that needs to be executed, ensure it has execute permissions for the service user.
chmod +x /usr/local/bin/my_script.sh
Why it works:
The systemd process, running under a specific user, must have the necessary read, write, or execute permissions on the target file or directory to perform the action specified in ExecStartPre.
4. SELinux or AppArmor Restrictions
Diagnosis:
Security modules like SELinux or AppArmor can prevent systemd from accessing files or executing commands, even if standard file permissions would allow it. The "No Such File or Directory" error might be a misleading symptom of a denial.
Command: Check audit logs for SELinux or AppArmor denials. For SELinux:
sudo ausearch -m avc -ts recent
For AppArmor:
sudo dmesg | grep 'apparmor'
Or check journalctl:
sudo journalctl -xe | grep 'DENIED'
Fix:
If SELinux is the culprit, you’ll need to adjust its context or policy. For example, if your service needs to write to /opt/my_app/data, but SELinux is preventing it:
sudo semanage fcontext -a -t 'var_lib_t' '/opt/my_app/data(/.*)?'
sudo restorecon -Rv /opt/my_app/data
For AppArmor, you might need to edit the profile for the service or the application it’s running.
sudo vim /etc/apparmor.d/your_service_profile
And add the necessary path. Reload AppArmor: sudo systemctl reload apparmor.
Why it works:
SELinux and AppArmor enforce mandatory access control policies that can override standard Unix permissions. Correcting these policies or contexts allows the systemd process to access the required resources.
5. Typo in the .service file itself
Diagnosis:
A simple, but common, mistake is a typo in the .service file when specifying the ExecStartPre command or its arguments, leading systemd to interpret the string incorrectly as a file path.
Command:
Carefully review the ExecStartPre line in your .service file for any extra spaces, missing characters, or incorrect syntax.
sudo systemctl cat your_service_name.service
Fix: Correct the typo. For instance, if you accidentally typed:
ExecStartPre=/usr/local/bin/my_script.sh --config /etc/my_app/config.yml
but meant:
ExecStartPre=/usr/local/bin/my_script.sh --config /etc/my_app/config.yml
(e.g., a misplaced space that systemd interprets as part of the file name), fix it. After editing, always reload the systemd daemon:
sudo systemctl daemon-reload
And then restart your service:
sudo systemctl restart your_service_name.service
Why it works:
systemd parses the .service file literally. Any syntax error or incorrect character in the ExecStartPre directive can cause it to misinterpret the command or file path it’s supposed to execute.
6. File System Mounted Read-Only or Unavailable
Diagnosis:
The file or directory that ExecStartPre is trying to access is on a file system that is mounted read-only, or the file system itself is not available or properly mounted.
Command: Check the mount status of the relevant file system.
mount | grep '/path/to/filesystem'
Look for ro (read-only) flags or if the path is listed at all.
Fix:
Remount the file system with write permissions or ensure it’s mounted correctly at boot.
If /data is mounted read-only:
sudo mount -o remount,rw /data
If the file system isn’t mounting automatically, check /etc/fstab for the correct entry and ensure the device is healthy.
Why it works:
ExecStartPre might need to create, modify, or delete files. If the underlying file system is mounted read-only or is inaccessible, these operations will fail, and systemd might report a "No Such File or Directory" error if it’s trying to interact with a path that cannot be written to.
Once these are fixed, you’ll likely encounter issues related to the actual execution of your ExecStartPre script or command, such as missing dependencies, incorrect arguments, or application-specific configuration errors.