Medium · reference design
Distributed job scheduler
Run millions of one-off and recurring jobs at the right time: never lose one, never fire the same run twice by accident, retry what fails, and keep working when machines die.
1 Requirements
Functional
- Schedule a job to run once at a given time, or on a recurring schedule (cron).
- Run the job, retry it on failure, and give up after a limit.
- Cancel or pause a job, and see the history of every run.
Non-functional
- Durable: a scheduled job survives restarts and machine failures.
- Each run starts within a few seconds of its scheduled time.
- At-least-once execution, with job code written so a duplicate is harmless.
- Scale to tens of millions of jobs a day by adding machines.
Out of scope: Workflow orchestration with dependencies between jobs. The business logic inside each job.
2 Core entities
- Job
- Type, payload, schedule (time or cron plus time zone), next run time, status.
- Run
- One execution of a job: attempt number, worker, start, end, result.
- Partition lease
- Which dispatcher owns which slice of jobs, and until when.
3 API
POST /jobsBody: type, payload, runAt or cron + timeZone, maxAttempts, idempotencyKey. Returns 201 with the job id.
POST /jobs/{id}/pause · /jobs/{id}/cancelStops future runs; a run already in progress finishes.
GET /jobs/{id}/runsHistory of every attempt with status, duration and error.
4 Data flow
- A service schedules a job; it is stored with its next run time.
- Dispatchers, each owning a slice of jobs, claim the ones that are due and put them on a queue.
- Workers run them idempotently, report progress and record the result; failures are retried, then dead-lettered.
5 High-level design
Press play to follow one request through the system. Every step explains what happens and the trade-off behind the choice.
Follow one job from the moment it is scheduled until it has run, including what happens when a worker dies or the job keeps failing. Press play, or step through the stages.
6 Deep dives
How do we find due jobs at scale?
| Option | Gives | Costs |
|---|---|---|
| Poll an indexed table | Simple, transactional, easy to cancel or edit jobs. | Polling load grows with dispatchers; precision is the poll interval. |
| Sorted set in an in-memory store | Very fast range queries by time. | Durability and consistency with the job records become your problem. |
| Time-bucketed tables or delay queues | Scales to huge volumes; each bucket is read once. | Cancelling or rescheduling a job means finding it in the right bucket. |
Choice. Poll a partitioned, indexed table. It covers tens of millions of jobs a day; move hot partitions to time buckets only if polling becomes the bottleneck.
How do we stop a stale dispatcher or worker from acting after its lease expired?
| Option | Gives | Costs |
|---|---|---|
| Trust the lease timeout | Nothing extra. | A paused process can wake up after expiry and still write. |
| Fencing tokens | Each lease has an increasing number; writes with an older number are rejected. | Every write path has to check the token. |
Choice. Fencing tokens on leases and runs. A long garbage-collection pause or network partition can then delay work, but never cause a double write.
How do recurring schedules handle time zones?
| Option | Gives | Costs |
|---|---|---|
| Store cron in UTC | One clock, simple math. | "Every day at 9:00" drifts by an hour when daylight saving changes. |
| Store cron plus the job time zone | Runs at the local time people expect, all year. | Must handle the hour that does not exist and the hour that happens twice. |
Choice. Cron plus time zone, computing each next run in that zone and storing it as UTC. The skipped and repeated hours get explicit rules: run at the next valid time, and only once.
In one sentence
Durable jobs indexed by next run time, partitioned dispatchers that claim and advance atomically, a queue to idempotent workers with heartbeats, bounded retries into dead letters, and a history that answers "did it run?".