Skip to main content
Linux · lesson 4 of 7

Beginner

2

Intermediate

5

Also part of Linux

7
Open the Linux page →
Linux intermediate Lesson 4 of 7

systemd Service Supervision: Restart Limits and Why They Trip

A service configured to restart on failure crash-loops until systemd's own StartLimitBurst safety valve trips and refuses to restart it again — a real failure that DigitalOcean's syntax-focused systemd guides don't walk through.

By SkillByExample · Published

linux systemd systemctl services

Restart=on-failure sounds like a promise to keep a crashing service alive forever. It isn’t — and the safety valve that overrides it is the more interesting behavior to actually see trip.

A service that restarts itself into a wall

docker run -d --name systemd-lesson --privileged --cgroupns=host \
  -v /sys/fs/cgroup:/sys/fs/cgroup:rw jrei/systemd-ubuntu:22.04
docker exec systemd-lesson ps -p 1 -o pid,comm
    PID COMMAND
      1 systemd

A purpose-built systemd image, confirmed booting real systemd as PID 1 — a plain ubuntu:22.04 with the systemd package installed does not do this by default. The service under test deliberately crashes one second after every start:

# /usr/local/bin/crashy.sh
#!/bin/bash
echo "crashy service starting, pid $$"
sleep 1
echo "crashy service crashing now"
exit 1
# /etc/systemd/system/crashy.service
[Unit]
Description=A service that crashes on purpose

[Service]
ExecStart=/usr/local/bin/crashy.sh
Restart=on-failure
RestartSec=2
StartLimitIntervalSec=60
StartLimitBurst=3

[Install]
WantedBy=multi-user.target
docker exec systemd-lesson systemctl daemon-reload
docker exec systemd-lesson systemctl start crashy.service

The bug: three restarts, then systemd gives up

sleep 6
docker exec systemd-lesson journalctl -u crashy.service --no-pager
Sep 14 06:13:57 d0ee153fcae9 systemd[1]: /etc/systemd/system/crashy.service:8: Unknown key name 'StartLimitIntervalSec' in section 'Service', ignoring.
Sep 14 06:13:57 d0ee153fcae9 systemd[1]: Started A service that crashes on purpose.
Sep 14 06:13:57 d0ee153fcae9 crashy.sh[82]: crashy service starting, pid 82
Sep 14 06:13:58 d0ee153fcae9 crashy.sh[82]: crashy service crashing now
Sep 14 06:13:58 d0ee153fcae9 systemd[1]: crashy.service: Main process exited, code=exited, status=1/FAILURE
Sep 14 06:13:58 d0ee153fcae9 systemd[1]: crashy.service: Failed with result 'exit-code'.
Sep 14 06:14:00 d0ee153fcae9 systemd[1]: crashy.service: Scheduled restart job, restart counter is at 1.
Sep 14 06:14:00 d0ee153fcae9 systemd[1]: Stopped A service that crashes on purpose.
Sep 14 06:14:00 d0ee153fcae9 systemd[1]: Started A service that crashes on purpose.
Sep 14 06:14:00 d0ee153fcae9 crashy.sh[84]: crashy service starting, pid 84
Sep 14 06:14:01 d0ee153fcae9 crashy.sh[84]: crashy service crashing now
Sep 14 06:14:01 d0ee153fcae9 systemd[1]: crashy.service: Main process exited, code=exited, status=1/FAILURE
Sep 14 06:14:01 d0ee153fcae9 systemd[1]: crashy.service: Failed with result 'exit-code'.
Sep 14 06:14:03 d0ee153fcae9 systemd[1]: crashy.service: Scheduled restart job, restart counter is at 2.
Sep 14 06:14:03 d0ee153fcae9 systemd[1]: Stopped A service that crashes on purpose.
Sep 14 06:14:03 d0ee153fcae9 systemd[1]: Started A service that crashes on purpose.
Sep 14 06:14:03 d0ee153fcae9 crashy.sh[92]: crashy service starting, pid 92
Sep 14 06:14:04 d0ee153fcae9 crashy.sh[92]: crashy service crashing now
Sep 14 06:14:04 d0ee153fcae9 systemd[1]: crashy.service: Main process exited, code=exited, status=1/FAILURE
Sep 14 06:14:04 d0ee153fcae9 systemd[1]: crashy.service: Failed with result 'exit-code'.
Sep 14 06:14:06 d0ee153fcae9 systemd[1]: crashy.service: Scheduled restart job, restart counter is at 3.
Sep 14 06:14:06 d0ee153fcae9 systemd[1]: Stopped A service that crashes on purpose.
Sep 14 06:14:06 d0ee153fcae9 systemd[1]: crashy.service: Start request repeated too quickly.
Sep 14 06:14:06 d0ee153fcae9 systemd[1]: crashy.service: Failed with result 'exit-code'.
Sep 14 06:14:06 d0ee153fcae9 systemd[1]: Failed to start A service that crashes on purpose.

Two things happen here, not one. First, the very first line: Unknown key name 'StartLimitIntervalSec' in section 'Service', ignoring — that key belongs in [Unit], so this unit is actually running on systemd’s built-in default interval (10 seconds), not the 60 this file asked for. Second, RestartSec=2 means each crash-restart-crash cycle takes about 3 seconds, so three of them fit inside even that shorter 10-second window — restart counter climbs 1, 2, 3, and on the fourth crash systemd refuses: Start request repeated too quickly. Restart=on-failure was honored exactly 3 times, then overridden by the burst limit built to stop exactly this kind of loop.

docker exec systemd-lesson systemctl status crashy.service
x crashy.service - A service that crashes on purpose
     Loaded: loaded (/etc/systemd/system/crashy.service; disabled; vendor preset: enabled)
     Active: failed (Result: exit-code) since Mon 2026-09-14 06:14:06 UTC; 8s ago
    Process: 92 ExecStart=/usr/local/bin/crashy.sh (code=exited, status=1/FAILURE)
   Main PID: 92 (code=exited, status=1/FAILURE)

Sep 14 06:14:06 d0ee153fcae9 systemd[1]: crashy.service: Scheduled restart job, restart counter is at 3.
Sep 14 06:14:06 d0ee153fcae9 systemd[1]: Stopped A service that crashes on purpose.
Sep 14 06:14:06 d0ee153fcae9 systemd[1]: crashy.service: Start request repeated too quickly.
Sep 14 06:14:06 d0ee153fcae9 systemd[1]: crashy.service: Failed with result 'exit-code'.
Sep 14 06:14:06 d0ee153fcae9 systemd[1]: Failed to start A service that crashes on purpose.

Active: failed — and it stays this way. A plain systemctl start crashy.service from here gets refused with the same “repeated too quickly” error, because the restart-limit breach itself is the recorded failure, independent of whether the script would crash again.

The fix: clear the limit, then fix the actual bug

docker exec systemd-lesson systemctl reset-failed crashy.service
docker exec systemd-lesson systemctl status crashy.service
* crashy.service - A service that crashes on purpose
     Loaded: loaded (/etc/systemd/system/crashy.service; disabled; vendor preset: enabled)
     Active: inactive (dead)

reset-failed clears the failed flag and zeroes the restart counter — but the script still exits 1, so restarting now would just repeat the entire loop. The real fix is the script:

# /usr/local/bin/crashy.sh, fixed
#!/bin/bash
echo "crashy service starting, pid $$"
exec sleep 3600
docker exec systemd-lesson systemctl start crashy.service
docker exec systemd-lesson systemctl status crashy.service
* crashy.service - A service that crashes on purpose
     Loaded: loaded (/etc/systemd/system/crashy.service; disabled; vendor preset: enabled)
     Active: active (running) since Mon 2026-09-14 06:19:05 UTC; 2s ago
   Main PID: 149 (sleep)
      Tasks: 1 (limit: 9346)
     Memory: 188.0K
     CGroup: /docker/d0ee153fcae9c7b84dddadd31c3b12a2a81b8fbb61f6b18ec1f37409dd3af7a8/system.slice/crashy.service
             `-149 sleep 3600

Sep 14 06:19:05 d0ee153fcae9 systemd[1]: Started A service that crashes on purpose.
Sep 14 06:19:05 d0ee153fcae9 crashy.sh[149]: crashy service starting, pid 149

Active: active (running), a real Main PID still alive seconds later — no restart counter in sight, because there’s nothing to restart from.

Practice

1. Confirm StartLimitBurst is still tracked correctly even with the interval key misplaced. 
docker exec systemd-lesson systemctl show crashy.service -p StartLimitIntervalUSec -p StartLimitBurst
StartLimitIntervalUSec=10s
StartLimitBurst=3

StartLimitBurst=3 came through from [Service] as written, but StartLimitIntervalUSec shows systemd’s own default (10s), not the 60 the file asked for — exactly the mismatch the misplaced key produces. Moving StartLimitIntervalSec=60 into [Unit] is what would actually get the real 60-second window.

2. Confirm the restart counter itself, not just the visible status, was zeroed by reset-failed. 
docker exec systemd-lesson systemctl show crashy.service -p NRestarts
NRestarts=0

reset-failed clears this internal counter along with the failed flag — without it, a service that had already used up 2 of its 3 allowed restarts would trip the limit after just one more crash, not three.

3. Distinguish a service that's inactive from one that never existed. 
docker exec systemd-lesson systemctl status nonexistent.service
Unit nonexistent.service could not be found.

A unit that doesn’t exist fails outright with “could not be found” — it never gets to print a Loaded/Active block at all. Compare that to crashy.service after reset-failed, which printed a full status with Loaded: loaded (/etc/systemd/system/crashy.service; ...) and Active: inactive (dead): the presence of a Loaded line is what actually confirms the unit file exists and is registered, regardless of whether it’s currently running.

Next: package management — installing, upgrading, and untangling a broken dependency with apt and dpkg.

Frequently Asked Questions

Why did systemd stop restarting a service that was configured to restart on failure?
Restart=on-failure only says systemd should restart the service after a crash — it doesn't promise to do that forever. StartLimitBurst caps how many restarts are allowed inside a rolling time window; once a service exceeds that count, systemd treats it as unrecoverable and marks it failed with 'Start request repeated too quickly', deliberately refusing further restarts.
Why does systemd have a restart-burst limit at all?
Without one, a service that crashes instantly on every start would restart in a tight loop forever, burning CPU and filling the journal with thousands of identical failure log lines. The burst limit turns an infinite crash loop into a single, visible 'failed' state after a few attempts, which is far easier to notice and alert on.
Does systemctl restart clear the exceeded restart-limit state?
No — the unit stays in the failed, refuses-to-start state until you explicitly run systemctl reset-failed, which clears both the failed flag and the restart counter for that unit. A plain systemctl start issued before reset-failed will just get refused the same way the automatic restart was.
Why did my StartLimitIntervalSec setting seem to be ignored?
It was in the wrong section. StartLimitIntervalSec and StartLimitBurst belong in [Unit], not [Service] — systemd 249 logs 'Unknown key name... in section Service, ignoring' and silently falls back to its built-in default interval (10 seconds) instead of erroring out. The burst count can still take effect from [Service] in some systemd versions, which makes the mistake easy to miss: the limit still trips, just on a different window than the one written in the file.
What's the difference between a service that's inactive and one that's failed?
inactive (dead) means the service isn't running and nothing is wrong — it either was never started or stopped cleanly. failed means its last run ended in an error systemd noticed (non-zero exit code, or a restart-limit breach), and that failure is recorded until something acknowledges it, either by starting the service successfully again or by running systemctl reset-failed.