Limits

Kaabist bounds how fast you may call the API, how many jobs run at once, and how much any one job may spend. Each limit has one answer you can program against: a 429 with Retry-After, a job that waits in the queue, or a job that fails with a code naming the limit it hit.

#When to use it

Read this page when you build a client that sends many requests, when you size a batch or a schedule, or when a response is 429, 403 plan_required or a job failed with timeout, max_steps_exceeded, max_credits_exceeded or max_tier_exceeded. The numbers per plan are on Plans and pricing.

#The layers

Only the first six ever refuse a request. Concurrency never does.

#Requests

Every API key has a bucket of requests. It holds a sixth of your plan's per-minute rate (at least ten) and refills evenly across the minute, so you can burst briefly and then continue at the rate:

All of your organisation's keys also share a bucket three times that size, so more keys do not mean more requests. A request costs one token; POST /v1/jobs/batch costs five per started hundred items. Every response tells you where you stand:

X-RateLimit-Limit: 60
X-RateLimit-Remaining: 7
X-RateLimit-Reset: 1790503261

Limit is requests a minute for the bucket that judged the request (the key's for an API key, the organisation's for a panel session), and Reset is the Unix second at which that bucket is full again. Past the limit you get 429 rate_limited with Retry-After in seconds and error.details.retry_after_seconds; error.details.scope says which bucket refused. Wait that long and try again.

#Jobs running at once

Each plan runs a number of jobs at once per tier, counted separately: using all your browser slots does not stop http jobs. GET /v1/org shows the limits and how many are running now:

"quotas": {
  "http": { "limit": 20, "in_use": 3 },
  "browser": { "limit": 4, "in_use": 4 },
  "stealth": { "limit": 1, "in_use": 0 }
}

This one never refuses anything. A job created while every slot is busy waits in queued and starts when one of your own jobs finishes; the order among waiting jobs is described in Queueing.

#Test keys

Jobs created with an sk_test_ key run for real and cost nothing. Instead, each plan allows a number of them a day, counted per UTC day; past it, creating one answers 429 test_quota_exceeded with Retry-After until midnight UTC and error.details holding limit, used and resets_at. Their results are kept for 24 hours.

// The key decides the mode: an sk_test_ key runs test jobs, which the live data never sees.
const probe = await kaab.jobs.create({
  url: `${site}/p/1`,
  extract: { title: "h1" },
  wait: 30,
});
console.log(`${probe.id} ran in ${probe.mode} mode`);

#Programs that sign in

A program that fills a password field or a secret input runs at most sixty times an hour per organisation; past that, 429 login_flow_rate_limited with Retry-After. It also needs a verified payment method.

#Limits on one job

Every job runs inside a budget. Each limit fails the job with its own error code, so you can tell which one it hit:

Time a job spends parked, waiting for input or for you, does not count toward its duration. A single step is also bounded by its own timeout_ms; a step that runs past its budget fails with timeout even when the job has time left.

max_tier is a ceiling, not a demand: a job that allows the browser tier but does not need it still runs at http. A cache option on a job that cannot be cached (it has inputs or a session) is ignored rather than refused.

#What your plan includes

Some requests are refused because of what they need rather than how often they come. These answer 403, with error.details.rule saying which rule applied and error.details.upgrade_url pointing at your billing page:

• A verified payment method (payment_method_required). The free trial runs the http tier only. Browser and stealth runs, evaluate, bring-your-own providers and programs that sign in need a payment method on file.
• A plan that sells it (plan_required). A tier above your plan's highest tier, evaluate, bring-your-own providers, and the us region are not part of every plan.

A schedule that fires more often than your plan allows is refused when you create it with 422 schedule_interval_too_small.

#Worked example

A script creates 500 jobs against a shop's product pages with a free-plan key: 60 requests a minute, a burst of 10.

1. The first ten POST /v1/jobs answer 202 at once; X-RateLimit-Remaining counts down from 9 to 0.
2. The eleventh answers:
   Copy

   HTTP/1.1 429 Too Many Requests
   Retry-After: 1
   X-RateLimit-Limit: 60
   X-RateLimit-Remaining: 0
   

   Copy

   { "error": { "code": "rate_limited", "message": "…", "details": { "scope": "key", "retry_after_seconds": 1 }, "request_id": "…" } }
   

3. The bucket refills one request a second, so from then on the script can send about one request a second.

Meanwhile the jobs themselves are a separate matter: the free plan runs one http job at a time, so the accepted jobs wait in queued and run one after another. That never produces a 429.

#What the SDKs do for you

The TypeScript, Python and Go SDKs share one retry policy, so a script written with them handles the example above without extra code:

• they retry on 408, 429, 500, 502, 503 and 504, on a connection failure and on a per-attempt timeout, and on nothing else;
• they wait for the response's Retry-After when it has one, and otherwise back off exponentially from half a second up to eight seconds, with jitter;
• a Retry-After longer than a minute is not waited for: the SDK raises the error and leaves the decision to you. A test_quota_exceeded that resets at midnight is the usual case;
• they retry at most twice by default (three attempts). Change it with maxRetries in TypeScript and max_retries in Python (on the client or on one call), or with WithMaxRetries on a Go client and WithRequestMaxRetries on one call;
• they retry only requests that are safe to repeat: GET, PUT, DELETE, and a POST that carries an Idempotency-Key. jobs.create and jobs.batch always send one, so a retried create never runs a job twice.

When the retries are used up, the error the SDK raises carries the API's code (rate_limited) and the wait the API asked for (retryAfter in TypeScript, retry_after in Python, RetryAfter in Go). For a large backlog, prefer POST /v1/jobs/batch: one request for up to 1 000 items costs five tokens per hundred.

#Pitfalls

• More keys do not mean more requests. The organisation's bucket is shared by every key.
• Do not poll in a tight loop. GET /v1/jobs/{id} with Prefer: wait=<seconds> waits for the job's next state change in one request, and webhooks cost no requests at all.
• A Retry-After past a minute is a quota, not congestion. Retrying sooner only gets the same answer.
• max_duration_ms only lowers the default. A browser program that needs longer than five minutes should be split, or wait on await_input where a person is involved.
• A job waiting for a concurrency slot is not stuck. queued with no meta.blocked_reason means it is waiting its turn.

#Reference

• Error codes, including every 429 and 403 above
• Get the organisation (its quotas)
• Create a batch
• Options: max_duration_ms, max_steps, max_credits, max_tier
• SDKs and the SDK guide
• Plans and pricing