The most surprising thing about systemd instantiated services is that they aren’t really "services" in the traditional sense at all; they’re templates that systemd uses to dynamically generate actual service units on the fly based on parameters you provide.
Imagine you need to run a web server for multiple virtual hosts, each on its own port. Instead of writing a separate .service file for apache@80.service, apache@81.service, apache@82.service, and so on, you write a single template file: apache@.service.
Here’s what that template might look like:
[Unit]
Description=Apache Web Server for %i
After=network.target
[Service]
Type=simple
ExecStart=/usr/sbin/apache2 -k start -D FOREGROUND -c "Listen %i"
ExecReload=/usr/sbin/apache2 -k graceful
ExecStop=/usr/sbin/apache2 -k stop
PIDFile=/run/apache2/apache2-%i.pid
User=www-data
Group=www-data
Restart=on-failure
[Install]
WantedBy=multi-user.target
Notice the %i in Description, Listen, and PIDFile. This is the placeholder. When you enable or start apache@80.service, systemd takes this template, replaces %i with 80, and creates a temporary, actual service unit named apache@80.service for you to manage. The same happens for apache@81.service, apache@82.service, etc.
This templating mechanism is incredibly powerful for managing resources that follow a common pattern but differ in a specific parameter. Think of database instances, mail servers, or any application that needs to bind to a unique port or operate on a distinct directory.
Let’s see it in action. Suppose we have a hypothetical application my-app that can be configured to run with different user IDs. We’d create a template file at /etc/systemd/system/my-app@.service:
[Unit]
Description=My Application Instance for User %i
After=network.target
[Service]
User=user%i
ExecStart=/usr/local/bin/my-app --user-id %i
Restart=on-failure
[Install]
WantedBy=multi-user.target
Now, to run an instance for user123, you’d execute:
sudo systemctl start my-app@123.service
To check its status:
sudo systemctl status my-app@123.service
And to see the actual, dynamically generated unit file that systemd created internally (you can’t edit this directly, it’s ephemeral):
systemctl cat my-app@123.service
This command will show you the expanded version of the template, with %i replaced by 123.
The core problem instantiated services solve is reducing configuration duplication. Instead of maintaining dozens of nearly identical .service files, you have one template. This makes updates and maintenance vastly simpler. If you need to change how my-app runs, you edit the single my-app@.service template, and all instantiated services derived from it will pick up the change upon restart.
The %i is the most common specifier, representing the "instance" part of the unit name. However, systemd supports other specifiers like %p (the base name of the unit, if the template is foo@bar.service and you start foo@baz.service, %p would be foo) and %f (a filesystem-path-like interpretation of the instance name, useful for constructing paths).
When you enable an instantiated service, like sudo systemctl enable my-app@123.service, systemd creates a symlink in /etc/systemd/system/multi-user.target.wants/ (or wherever WantedBy points) that points to the template file, but with the instance name appended. For example, /etc/systemd/system/multi-user.target.wants/my-app@123.service would be a symlink to ../my-app@.service. This tells systemd that my-app@123.service should be started when multi-user.target is reached.
The mechanism that often trips people up is how systemd resolves dependencies and ordering for these. While After=network.target in the template applies to all instances, if you have a dependency between two different instantiated services, like my-app@1.service needing to start after my-app@2.service, you cannot define this directly in the template. Instead, you’d need to define a separate, static service unit that declares these dependencies and then ensure both instantiated services are started by that static unit.
The other crucial aspect is error handling. If the template itself has a syntax error, systemctl commands targeting any instance will fail with a generic "invalid unit file" error, and it can be tricky to pinpoint the exact line in the template causing the issue. Always run systemd-analyze verify /etc/systemd/system/your-template@.service after making changes.
When you start or stop an instantiated service, systemd doesn’t just process the template once; it dynamically generates the unit configuration each time it needs to interact with that specific instance. This is why systemctl status my-app@123.service works even though my-app@123.service isn’t a static file on disk.
The next thing you’ll likely encounter is managing multiple instantiation specifiers, where a unit name might look like service@host1-port8080.service, and you need to parse both host1 and port8080 within the template.