Egress and proxies
How a job's traffic reaches a site: proxy pools, countries, your own proxy, pacing per site, what is refused before it leaves, and what is recorded.
Every request a job makes, from the http tier and from a real browser alike, leaves through one gateway that picks the exit, paces each site and refuses destinations that must not be reached. That is what lets Kaabist visit a site politely however many customers want it at once, bill proxy traffic exactly, and guarantee that a program cannot be tricked into reaching something internal.
#When to use it
The defaults suit public pages: a datacenter exit in the job's region, moved to residential when a site answers with a challenge. Set the proxy option when a site shows different content per country, when it blocks datacenter addresses, or when your contract requires traffic to leave through a proxy you operate. Read the refusal table when a job fails with ssrf_blocked, byo_proxy_unreachable or a navigation that never reached the site.
#Choosing an exit
The proxy option on a program or a job decides where a request comes from:
{ "options": { "proxy": { "mode": "platform", "pool": "datacenter", "country": "DE" } } }
poolisdatacenter(the default, and the fastest and cheapest),residential, ormobile. Kaabist moves up to residential on its own when a site answers with a challenge page, or when the site's profile says it needs one, and the stealth tier always uses residential.countryis an ISO-3166-1 alpha-2 code. If no provider in the chosen pool serves it, the nearest configured country is used rather than failing the job.mode: "byo"with abyo_idsends the job through your own proxy instead.fallbackthen decides what happens when yours is unreachable:"platform"keeps the job running on Kaabist's exits, adds abyo_fallback_usedwarning to the result and bills that traffic, and"none"(the default) fails the job withbyo_proxy_unreachablerather than sending your traffic through an exit you did not choose.mode: "byo"without abyo_iduses your organisation's default proxy. See Bring your own providers for vendors, presets and health checks.
One job keeps one exit for its whole run. A checkout flow that changed address halfway would sign itself out, so the choice is made once and stays; a retry after a failed connection may land on a different exit.
meta.proxy on the job reports what was used: { "mode": "platform", "pool": "datacenter", "country": "DE" }.
#What the pools cost
Proxy traffic is billed per megabyte the gateway moved for the job: 0.05 credits for datacenter, 0.5 for residential, 2 for mobile. Traffic through your own proxy is not billed. See Credits and billing.
#Pacing
Kaabist limits how fast any one site is visited, across every customer. By default that is two requests per second to a hostname, with a burst of five; a site known to tolerate more, or less, gets its own limit. A request that arrives when the allowance is spent waits its turn, up to twenty seconds, and is then refused as rate limited rather than queued forever. A site also has a cap on connections open to it at once, across all customers.
This is why a job against a large catalogue takes the time it takes. It is also why Kaabist is blocked less often than a scraper that does not pace itself.
#What Kaabist has learned about a site
GET /v1/sites/{hostname} shows what the platform has learned about a site you use:
{ "hostname": "shop.example.com", "object": "site_profile", "tier_required": "browser", "tier_confidence": 0.9, "known_obstacles_count": 2, "api_endpoints_count": 1, "last_blocked_at": "2026-09-30T08:12:00Z" }
The tier it needs and how confident Kaabist is, how many obstacles it has learned to dismiss there, how many JSON endpoints were found behind it, and when it last blocked a request. It is the answer to "why is this site slower and more expensive than that one". A hostname nothing is known about returns an empty profile, not an error. What your own proxy or model taught Kaabist stays in your organisation's profile and never reaches the shared one.
#When a request never leaves
Some requests are refused before a connection is opened:
| Reason | What happened | What the job sees |
|---|---|---|
ssrf_blocked | The address the hostname resolves to is private, loopback, link-local or a cloud metadata endpoint, or the name ends in .internal or .local. This is not configurable: a program that could reach those could read Kaabist's own infrastructure. | fails with ssrf_blocked |
blocked_list | The target is on Kaabist's blocklist, or on a blocklist your organisation configured. | the navigation fails |
rate_limited | The site's allowance stayed spent for the whole twenty-second wait. | the navigation fails |
org_concurrency | Your organisation already has as many connections open as it may. | the navigation fails |
host_concurrency | The site already has as many connections open as Kaabist allows it, across all customers. | the navigation fails |
no_provider | The requested pool has no provider configured at all. | the navigation fails |
upstream_error | Two different proxy providers failed to open the connection. | the navigation fails |
byo_proxy_unreachable | Your own proxy could not be reached and fallback is none. | fails with byo_proxy_unreachable, not retryable; error.details.byo_id names the proxy |
A redirect is re-checked at every hop, and the address a hostname resolves to is checked and then connected to directly, so a page that redirects to an internal address, or a name that resolves to a public address once and a private one the next time, is refused rather than followed.
#What Kaabist records
For each connection: the hostname, the bytes in each direction, how long it lasted, and which pool it used. Not the URL path, not the headers, and not the content: a secure connection is opaque to the gateway by design. Bytes are what proxy usage is billed on. See Data retention for how long the rest of a job's data is kept.
#Worked example
A redirect to an internal address. A shop's link https://shop.example.com/go/partner answers with a 302 to http://169.254.169.254/latest/meta-data/, the cloud metadata address. A job that opens it reaches the shop, receives the redirect, and the gateway refuses the second hop: the job fails with error.code: "ssrf_blocked". The same happens with a redirect to 127.0.0.1. No setting changes this.
A site that blocks datacenter addresses. Some sites let only residential exits through. A job with the default proxy reaches it from a datacenter address and receives a challenge page; Kaabist recognises the challenge, moves to a residential exit and a stronger tier as your max_tier allows, and the job's meta.proxy.pool and meta.escalations show what it took. The residential traffic is billed at 0.5 credits a megabyte instead of 0.05.
Your own proxy, with a fallback. With a proxy registered as byo_01J…:
{ "options": { "proxy": { "mode": "byo", "byo_id": "byo_01J…", "fallback": "platform" } } }
If your proxy is down when the job runs, the job continues through Kaabist's exits, its result carries { "code": "byo_fallback_used" }, and that traffic is billed. With "fallback": "none" the same job fails with byo_proxy_unreachable and nothing leaves through an exit you did not choose.
#Pitfalls
- A country nobody serves is replaced, not refused. When the content depends on the country, test the job from that country before relying on it in a schedule.
- Residential costs ten times datacenter. Pin
pool: "datacenter"only when you are sure the site accepts it; otherwise let Kaabist move up when it must. fallback: "platform"is billed. The fallback traffic goes through Kaabist's pools and is charged like any other.- Pacing is per site, not per job. Splitting a crawl into more jobs does not make one site answer faster.
- Development addresses are refused in production. A program that works against
localhoston your machine is refused the same address on Kaabist.