The systemd-tmpfiles service is failing to clean up temporary files because its HoldOff timer is set too aggressively, preventing it from recognizing that files older than the configured MaxAge should be removed.
This is a common scenario where the systemd-tmpfiles daemon, responsible for managing temporary files and directories, gets stuck in a loop where it’s configured to remove files older than a certain age, but it’s also configured with a HoldOff time that’s too short. The HoldOff time is essentially a grace period after a file is created before systemd-tmpfiles will even consider removing it. If HoldOff is shorter than the interval at which systemd-tmpfiles runs, it might never get a chance to evaluate files for removal before they become "too new" according to the HoldOff setting.
Here are the most common reasons this happens and how to fix them:
-
systemd-tmpfilesHoldOffinterval is too short relative to its execution frequency.- Diagnosis: Check the
systemd-tmpfilesconfiguration. The primary configuration file is/etc/tmpfiles.conf, and additional configurations can be found in/usr/lib/tmpfiles.d/*.conf. Look for theHoldOffdirective. The default is usually0(meaning no hold-off), but it can be set to a specific duration (e.g.,1dfor one day). Also, check the timer unit forsystemd-tmpfiles-clean.timer(or similar) to see how often it runs.systemctl cat systemd-tmpfiles-clean.timer grep HoldOff /etc/tmpfiles.conf /usr/lib/tmpfiles.d/*.conf - Fix: If
HoldOffis set to a value greater than0and it’s causing issues, the simplest fix is often to set it to0or a very small value like10s. This tellssystemd-tmpfilesto consider files for removal immediately after theirMaxAgehas passed, without a grace period.
This works because setting# Edit /etc/tmpfiles.conf or a file in /usr/lib/tmpfiles.d/ # Change or add the line: HoldOff=0HoldOff=0removes the grace period, allowingsystemd-tmpfilesto act on files as soon as theirMaxAgeis exceeded, regardless of how frequently the timer runs. - Why it works: The
systemd-tmpfilesdaemon operates on a schedule. When it runs, it looks at files and directories defined in its configuration. For each entry, it checks two primary conditions:MaxAge(how old a file can be before it’s eligible for deletion) andHoldOff(how long after creation a file is immune from deletion, even ifMaxAgeis met). IfHoldOffis, say,1dandMaxAgeis30d, a file created today won’t be considered for deletion until it’s 1 day old, and then it will be deleted if it’s older than 30 days. If the timer runs every hour, andHoldOffis set to24h, the file only gets a chance to be evaluated for deletion after it’s 24 hours old. If the timer runs less frequently than theHoldOffduration, it might never get a chance to evaluate files for removal. SettingHoldOff=0ensures that the moment a file’s age exceedsMaxAge, it becomes eligible for deletion during the nextsystemd-tmpfilesrun.
- Diagnosis: Check the
-
Incorrect
MaxAgecombined withsystemd-tmpfilesexecution frequency.- Diagnosis: If
HoldOffis0or not set, but files aren’t being cleaned, theMaxAgemight be set to an extremely long duration (e.g.,1yfor one year) or a value that is never realistically reached by the files in question. The timer might also be running less frequently than you expect.grep MaxAge /etc/tmpfiles.conf /usr/lib/tmpfiles.d/*.conf systemctl cat systemd-tmpfiles-clean.timer - Fix: Adjust
MaxAgeto a more appropriate value for the temporary files you’re managing. If you want files cleaned daily, setMaxAge=1d. If you want them cleaned hourly, setMaxAge=1h.
This works by directly setting the threshold for deletion. If the files are expected to be temporary and should not persist beyond a certain time, setting# Edit a relevant .conf file in /usr/lib/tmpfiles.d/ # Example for cleaning /var/log/myapp/tmp: d /var/log/myapp/tmp 0755 root root 1dMaxAgeto that duration ensures they are marked for deletion once they reach that age. - Why it works:
MaxAgeis the hard limit for how long a file can exist. If you haveMaxAge=1yand yourHoldOffis0, files will only be considered for deletion once they are a year old. If you intended for them to be cleaned up daily, you need to setMaxAge=1d.
- Diagnosis: If
-
Permissions issues preventing
systemd-tmpfilesfrom accessing or deleting files.- Diagnosis:
systemd-tmpfilesruns as root. If the temporary files or directories are owned by a different user or group and lack the necessary write permissions for root,systemd-tmpfileswon’t be able to delete them. Check the ownership and permissions of the directories where temporary files reside.ls -ld /path/to/temporary/directory ls -l /path/to/temporary/directory/some_file - Fix: Ensure that root has write permissions on the parent directories of the temporary files. The simplest way is often to ensure the directory itself is owned by root and has appropriate permissions (e.g.,
755).chown root:root /path/to/temporary/directory chmod 755 /path/to/temporary/directory - Why it works: The
systemd-tmpfilesprocess runs with root privileges. If the directory containing the temporary files is not writable by root, theunlink()system call (used for deletion) will fail with aEACCES(Permission denied) error, even thoughsystemd-tmpfilesis technically running as root. Correcting ownership and permissions ensures that root can perform the delete operation.
- Diagnosis:
-
SELinux or AppArmor restrictions.
- Diagnosis: Security Enhanced Linux (SELinux) or AppArmor can prevent processes, even root, from performing certain actions. If
systemd-tmpfilesis trying to delete files in a location that SELinux or AppArmor policy doesn’t permit, it will fail silently or with an audit log entry.# For SELinux: sudo ausearch -m avc -ts recent # For AppArmor: sudo dmesg | grep -i apparmor sudo journalctl -xe | grep -i apparmor - Fix: Adjust SELinux or AppArmor policies to allow
systemd-tmpfilesto manage the specific directories and file types. This is often complex and requires understanding the specific security context. A temporary workaround for testing is to set SELinux to permissive mode:
For AppArmor, you might need to edit profiles in# For SELinux (temporary, use with caution): sudo setenforce 0 # Then re-enable: sudo setenforce 1/etc/apparmor.d/. - Why it works: SELinux and AppArmor enforce mandatory access control. If the policy dictates that the
systemd-tmpfilesprocess (or its context) cannot write/delete files in a particular directory, it will block the operation. Modifying the policy to grant the necessary permissions resolves the conflict.
- Diagnosis: Security Enhanced Linux (SELinux) or AppArmor can prevent processes, even root, from performing certain actions. If
-
Filesystem issues or full disk.
- Diagnosis: If the filesystem is full,
systemd-tmpfiles(or any process) cannot create new files, but it also might encounter issues during deletion if metadata operations fail. Check disk space and filesystem health.df -h sudo fsck /dev/sdXN # Replace /dev/sdXN with your partition - Fix: Free up disk space or expand the partition. If there are filesystem errors, run
fsck(ensure the partition is unmounted or in read-only mode).# Example: sudo apt autoremove # Or similar package cleanup sudo rm -rf /path/to/large/unneeded/files - Why it works: Deletion involves writing to the filesystem’s metadata to mark blocks as free. If the filesystem is full or corrupted, these metadata operations can fail, preventing successful deletion.
- Diagnosis: If the filesystem is full,
-
System clock drift or incorrect time.
- Diagnosis: While less common for
systemd-tmpfileswith itsMaxAgeandHoldOffdirectives which are relative to file mtime, significant clock drift could theoretically cause issues ifsystemd-tmpfilesrelies on absolute timestamps in some edge cases or if other related cleanup mechanisms are involved.date timedatectl status - Fix: Ensure the system clock is synchronized using NTP.
sudo timedatectl set-ntp true - Why it works: File modification times (
mtime) are recorded relative to the system’s clock. If the clock is wildly inaccurate, the calculation of a file’s age could be incorrect, leadingsystemd-tmpfilesto misjudge whether a file has exceeded itsMaxAge.
- Diagnosis: While less common for
After addressing these, you might encounter the next common issue: the systemd-journald service potentially growing too large if its logs are not being rotated or purged correctly.