Queueing and fairness

Creating a job does not start it: the job is written down, a dispatcher picks it up, usually within a quarter of a second, and a worker runs it. This page is what happens in between, because "why has my job not started yet" has a short list of answers, and only one of them needs you to do something.

#When to use it

Read this page when a job stays queued longer than you expect, when you see meta.blocked_reason or receive a job.blocked event, when you submit large batches next to latency-sensitive jobs, or when you want to know what a worker crash means for a job that was halfway through.

#Why a job is waiting

Only the credits case needs you to act. The job.blocked event fires once when a job enters that state, with reason: "insufficient_credits", not on every attempt to dispatch it. meta.blocked_reason is cleared when the job is dispatched. A job waiting for a session lock is served in the order the jobs asked for it, and fires no event.

#Your turn is weighted, not first-come-first-served

If one customer submits fifty thousand jobs, everybody else's jobs still start in seconds. Each organisation has a share of the queue proportional to its plan's weight (1 on free, 2 on developer, 4 on team, 8 on scale), and dispatching a job spends that share in proportion to what the job is expected to cost: a stealth-tier run spends twenty times what an http-tier one does.

Two consequences:

• Idleness is not saved up. An organisation that has been quiet for an hour does not get to empty the queue when it wakes up; it rejoins at the current position and takes its share from there.
• A big batch does not lock anyone out, including you. Your own five thousand jobs are spread across rounds rather than dispatched in one go, so a job you submit afterwards is not stuck behind all of them.

priority (0 to 9 on job creation) sits above all of this: a priority-9 job goes before every priority-0 job, whatever the weights say. Fairness then decides the order inside each priority band.

#Concurrency is per plan and per tier

Your plan buys a number of concurrent jobs at each tier, and the three are counted separately: using all your browser slots does not stop http jobs from running. A job over the limit stays queued and starts as soon as one of your own jobs at that tier finishes; it is never failed for it. The numbers per plan are on Plans and pricing.

#Credits are reserved, then settled

Creating a job reserves what it might cost; finishing it settles what it did cost and gives the rest back. If the reservation cannot be made at creation, the request answers 402 insufficient_credits and no job exists. A job that fails because of an internal error is refunded in full. Time a job spends parked is free.

Some jobs need to reserve again after they were created, and that is when one can wait as insufficient_credits instead of failing. An escalation is the common case: a job that starts at the http tier and finds a page that needs a browser goes back to the queue asking for a browser, and must reserve the difference. See Credits and billing.

#What happens when a worker dies

A running job holds a lease that its worker renews every ten seconds. If the worker is replaced, crashes or is killed, the lease expires, and within fifteen seconds of that the job goes back to the queue and restarts from its last checkpoint, without reserving its credits twice. You see this as a job that took longer, not as a job that failed. Its session lock, if it had one, is released at the same moment.

There is one exception, and it is deliberate. If the worker stopped in the middle of a step marked side_effect: true (placing an order, sending a message), nobody can know whether it happened. The job is not retried. It parks as needs_attention with the reason side_effect_ambiguous, and a person decides. Kaabist would rather ask you than place an order twice. See Side-effect safety.

A job always ends: a step that runs past its own budget, or a job past its max_duration_ms, is stopped by the worker's watchdog and fails with timeout.

#Cancelling

POST /v1/jobs/{id}/cancel on a job that is queued or parked cancels it at once and releases the credits it reserved. On a running job it answers with the job still running and the cancel recorded; the worker hears it at once and stops the run within seconds, and a job whose worker has not ended it thirty seconds after the request is ended for it. A job inside an open side-effect fence cannot be canceled (409 fence_open), for the same reason it is not retried.

#Resuming a parked job

POST /v1/jobs/{id}/input and POST /v1/jobs/{id}/resume put a parked job back in the queue, and the dispatcher sends it to a worker again. Both reserve credits first, and answer 402 if they cannot. When the value arrives while the browser context is still alive (within the step's keep-alive window, five minutes by default), the job goes straight back to the worker holding that context and carries on from exactly where it stopped. After that it resumes from its last checkpoint instead, which is slower but produces the same result. meta.resumed_from says which happened.

#Worked example

Blocked on credits. An organisation runs a program against https://app.example.com/p/1, whose server prints a placeholder price, with prefer_tier: "http" and max_tier: "browser". The http fetch reads the server's placeholder, the program's expect on the price fails, and the job must escalate to a browser, which costs five credits instead of one. The organisation's available balance is two.

The job goes back to queued, and GET /v1/jobs/{id} shows:

{
  "id": "job_01J…",
  "status": "queued",
  "meta": { "tier_requested": "http", "blocked_reason": "insufficient_credits" }
}

A webhook endpoint subscribed to job.blocked receives one event; its type and data are:

{
  "type": "job.blocked",
  "data": { "job": "job_01J…", "program": "ssr-price@1", "reason": "insufficient_credits", "url": "https://api.kaab.ist/v1/jobs/job_01J…" }
}

Someone buys a pack on the billing page. The dispatcher's next round finds the balance can cover the reservation, the job is dispatched, blocked_reason disappears, and the job finishes at the browser tier with an escalated warning and meta.escalations recording the move.

Requeued after a worker was replaced. A long browser job is running when its worker is replaced during a deployment. The job's Events tab in the panel shows started, then lease_expired once the lease runs out, then leased and started again as another worker picks it up from its last checkpoint, then finished. The result is the same as an uninterrupted run, it took longer, and it was not charged twice.

#Pitfalls

• queued with no blocked_reason is not a problem. It is your concurrency limit or the queue working as sold. Raise concurrency with a bigger plan, not with more API keys.
• A batch shares your concurrency with everything else. Give latency-sensitive jobs a higher priority instead of waiting for the batch to drain.
• An exclusive session serialises its jobs. Ten jobs on one exclusive session run one at a time. Mark the session shared only when the site tolerates parallel use; see Sessions.
• Every side-effect step should be followed by a checkpoint. A requeued job restarts from its last checkpoint; without one after the side effect, a crash parks the job for a person.
• Out of credits pauses schedules too. They resume when credits arrive, and they do not fire while paused.

#Reference

• Get a job, cancel a job, resume a job
• Get the organisation (its quotas)
• Job statuses and attention reasons
• The job.blocked, job.queued and job.started events
• Limits and Credits and billing