DEV Community

delay.run
delay.run

Posted on Edited on

How late is 'every minute'? An experiment you can run on cron, systemd timers and sleep loops

Reliable HTTP cron and delayed-job scheduling with automatic retries and full run logs — delay.run

"Run this every minute" sounds precise. It isn't. Each scheduler you can reach for on Linux has its own idea of what "every minute" means, and the differences matter when a job has to line up with an external deadline, a rate limit window, or another job.

Instead of quoting someone else's numbers (which depend on hardware, load and configuration), this post gives you a reproducible experiment: three schedulers, one logging script, and a tiny analysis script. Run it for an hour and you'll have real data from your own box.

What we're measuring

For each firing we record the wall-clock time with millisecond precision, then compute:

  • Offset: how far after the intended second (e.g. :00) the job started.
  • Jitter: how much that offset varies from run to run.
  • Drift: whether the offset grows over time.

The logger

Every scheduler calls the same script:

#!/usr/bin/env bash
# /usr/local/bin/stamp.sh  usage: stamp.sh <label>
printf '%s %s\n' "$1" "$(date +%s.%3N)" >> /tmp/sched-drift.log
Enter fullscreen mode Exit fullscreen mode

chmod +x /usr/local/bin/stamp.sh. Keep it trivial, so the script's own startup time stays negligible next to what we're measuring.

Note for Alpine and BusyBox users: %3N (milliseconds) is a GNU date extension. BusyBox date doesn't support %N, so you'd get no milliseconds and the analysis would be meaningless. Check with date +%s.%3N: if the output doesn't end in three digits, install GNU coreutils (apk add coreutils) or swap the date call for python3 -c 'import time; print(f"{time.time():.3f}")'.

Scheduler 1: cron

# crontab -e
* * * * * /usr/local/bin/stamp.sh cron
Enter fullscreen mode Exit fullscreen mode

Scheduler 2: a systemd timer (default settings)

# /etc/systemd/system/stamp.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/stamp.sh systemd-default

# /etc/systemd/system/stamp.timer
[Timer]
OnCalendar=*-*-* *:*:00
[Install]
WantedBy=timers.target
Enter fullscreen mode Exit fullscreen mode

Scheduler 3: a systemd timer with AccuracySec=1s

Copy both files to stamp-precise.service and stamp-precise.timer. Give the service its own label, otherwise its runs are logged as systemd-default and the comparison collapses, and add one line to the [Timer] section:

# /etc/systemd/system/stamp-precise.service
[Service]
Type=oneshot
ExecStart=/usr/local/bin/stamp.sh systemd-precise

# /etc/systemd/system/stamp-precise.timer
[Timer]
OnCalendar=*-*-* *:*:00
AccuracySec=1s
[Install]
WantedBy=timers.target
Enter fullscreen mode Exit fullscreen mode

The AccuracySec=1s line is the interesting variable. systemd timers default to AccuracySec=1min, which allows systemd to delay a firing by up to a minute so it can batch wake-ups and save power. That's documented behavior, and plenty of people find out about it the hard way.

If a sleep loop or local cron drifts, you may not notice until the job is already late. delay.run runs the HTTP schedule for you and records each start, so “every minute” is something you can audit.

Scheduler 4: a naive sleep loop

# loop.sh: the "I'll just write a loop" approach
while true; do
  /usr/local/bin/stamp.sh loop
  sleep 60
done
Enter fullscreen mode Exit fullscreen mode

Run it with nohup ./loop.sh &. Notice the bug that's built in: sleep 60 starts after the work finishes, so every iteration takes 60 seconds plus however long the work took.

Enable everything:

sudo systemctl daemon-reload
sudo systemctl enable --now stamp.timer stamp-precise.timer
Enter fullscreen mode Exit fullscreen mode

Now wait an hour. Make tea.

The analysis

# analyze.py
from collections import defaultdict
from statistics import mean, pstdev

runs = defaultdict(list)
for line in open("/tmp/sched-drift.log"):
    label, ts = line.split()
    runs[label].append(float(ts))

print(f"{'scheduler':<18}{'n':>4}{'mean off':>10}{'stdev':>8}{'max':>8}{'first':>8}{'last':>8}")
for label, ts in sorted(runs.items()):
    offs = [t % 60 for t in ts]        # seconds after :00
    print(f"{label:<18}{len(ts):>4}{mean(offs):>10.3f}{pstdev(offs):>8.3f}"
          f"{max(offs):>8.3f}{offs[0]:>8.3f}{offs[-1]:>8.3f}")
Enter fullscreen mode Exit fullscreen mode

For the sleep loop, "seconds after :00" isn't meaningful at first, because it started at an arbitrary second. Look at first vs last: if last keeps creeping forward, that's drift. A more direct measure is the average gap between runs minus the intended interval, mean(ts[i] - ts[i-1]) - 60, which shows the overrun per iteration.

What to look for in your results

I won't hand you numbers to copy, because yours will depend on your kernel, virtualization and load. These are the patterns to check for, and they follow from how each scheduler is designed:

  • cron should land within a small fraction of a second after :00, fairly consistently. Classic cron daemons wake at the top of each minute, so the offset is mostly process start-up time.
  • systemd with default AccuracySec will likely show a large mean offset with a small stdev. systemd doesn't fire at a random point in the one-minute window each time: it picks a point that is specific to the host but stable, so all timers on the machine can wake together. Expect the offset to be consistently late by the same amount, not smeared across the minute.
  • systemd with AccuracySec=1s should behave much like cron.
  • The sleep loop will drift. Each iteration adds the script's run time, so the offset walks forward steadily. Over a day the job no longer runs "every minute", it runs every minute-and-a-bit.

If you want to see drift more clearly, add sleep 2 inside stamp.sh for the loop variant only and re-run. The walk becomes obvious within minutes.

Why this matters in practice

Rate-limited APIs. If a vendor allows 60 calls per minute per window and your job fires at :59.8 one minute and :00.3 the next, you can exceed the window without ever running "more than once a minute".

Jobs chained by time. "Export runs at 02:00, import runs at 02:05" assumes the export takes under five minutes and starts on time. Drift and jitter eat that margin.

Billing and reporting cutoffs. A job meant to run "at midnight" that fires at 00:00:45 might process events from both days, depending on how it queries.

Takeaways

  1. If you use systemd timers for anything time-sensitive, set AccuracySec explicitly. The default is tuned for power saving, not precision.
  2. Never use a sleep loop for a fixed schedule. If you must loop, sleep until the next boundary (sleep $((60 - $(date +%s) % 60))), not for a fixed duration.
  3. Don't chain jobs by clock time. Trigger the second job when the first one finishes, or have the first one schedule the second.

That last point is where a delayed-job service such as delay.run fits: instead of guessing "five minutes should be enough", the first job finishes and schedules the next HTTP call for exactly when it's needed. Whatever tool you use, events beat clock guesses.

Cleanup

sudo systemctl disable --now stamp.timer stamp-precise.timer
crontab -e   # remove the line
pkill -f loop.sh
Enter fullscreen mode Exit fullscreen mode

Related reading

Run the experiment and post your results in the comments: which scheduler surprised you? I'm especially curious whether anyone sees large systemd offsets on cloud VMs compared with bare metal.

Top comments (2)

Collapse
 
c-wind profile image
c-wind •

The pattern to expect from AccuracySec is a bit different from "offsets spread across many seconds." systemd places the elapse at a host-specific but stable point inside the window that starts at OnCalendar and ends AccuracySec later. That is lateness, not an early fire, and the point stays put across runs. On an idle box I would expect a large mean offset and a small stdev: the same late second every minute, not a smear. A busy host can still shove it when timers coalesce, which is why both columns in the table matter. If stdev is tiny and mean off is tens of seconds, the timer is doing what AccuracySec is for, not wandering.

For the sleep loop, first-vs-last of t % 60 hides slow drift. The mean of (ts[i] - ts[i-1]) minus 60 shows the per-iteration overrun directly, including the sub-second case before it walks across a whole second.

Two copy hazards: date +%s.%3N needs GNU date. BusyBox leaves %N empty, float() throws, and the hour-long log is unreadable. And the two systemd units have to pass different labels into stamp.sh. Renaming only the unit file still logs both as systemd-default, so the AccuracySec comparison collapses.

Collapse
 
delayrun profile image
delay.run •

Thanks, all three are fair. You're right about AccuracySec: systemd picks a stable, per-host point in the window, so a large mean with a tiny stdev is the expected result, not wandering, and "spread across many seconds" was the wrong way to put it. The interval-delta check is a better drift metric than t % 60, and the label is a real bug: the stamp-precise unit has to call stamp.sh with its own label or the comparison collapses. I'll update the post with the label fix and a GNU date note for BusyBox users.