# Egress and proxies

How a job's traffic reaches a site: proxy pools, countries, your own proxy, pacing per site, what is refused before it leaves, and what is recorded.

Every request a job makes, from the http tier and from a real browser alike, leaves through one gateway that picks the exit, paces each site and refuses destinations that must not be reached. That is what lets Kaabist visit a site politely however many customers want it at once, bill proxy traffic exactly, and guarantee that a program cannot be tricked into reaching something internal.

## When to use it

The defaults suit public pages: a datacenter exit in the job's region, moved to residential when a site answers with a challenge. Set the `proxy` option when a site shows different content per country, when it blocks datacenter addresses, or when your contract requires traffic to leave through a proxy you operate. Read the refusal table when a job fails with `ssrf_blocked`, `byo_proxy_unreachable` or a navigation that never reached the site.

## Choosing an exit

The `proxy` option on a program or a job decides where a request comes from:

```json
{ "options": { "proxy": { "mode": "platform", "pool": "datacenter", "country": "DE" } } }
```

- **`pool`** is `datacenter` (the default, and the fastest and cheapest), `residential`, or `mobile`. Kaabist moves up to residential on its own when a site answers with a challenge page, or when the site's profile says it needs one, and the stealth tier always uses residential.
- **`country`** is an ISO-3166-1 alpha-2 code. If no provider in the chosen pool serves it, the nearest configured country is used rather than failing the job.
- **`mode: "byo"`** with a `byo_id` sends the job through your own proxy instead. `fallback` then decides what happens when yours is unreachable: `"platform"` keeps the job running on Kaabist's exits, adds a `byo_fallback_used` warning to the result and bills that traffic, and `"none"` (the default) fails the job with `byo_proxy_unreachable` rather than sending your traffic through an exit you did not choose. `mode: "byo"` without a `byo_id` uses your organisation's default proxy. See [Bring your own providers](https://docs.kaab.ist/guides/byo-providers) for vendors, presets and health checks.

One job keeps one exit for its whole run. A checkout flow that changed address halfway would sign itself out, so the choice is made once and stays; a retry after a failed connection may land on a different exit.

`meta.proxy` on the job reports what was used: `{ "mode": "platform", "pool": "datacenter", "country": "DE" }`.

### What the pools cost

Proxy traffic is billed per megabyte the gateway moved for the job: 0.05 credits for datacenter, 0.5 for residential, 2 for mobile. Traffic through your own proxy is not billed. See [Credits and billing](https://docs.kaab.ist/concepts/credits-and-billing).

## Pacing

Kaabist limits how fast any one site is visited, across every customer. By default that is two requests per second to a hostname, with a burst of five; a site known to tolerate more, or less, gets its own limit. A request that arrives when the allowance is spent waits its turn, up to twenty seconds, and is then refused as rate limited rather than queued forever. A site also has a cap on connections open to it at once, across all customers.

This is why a job against a large catalogue takes the time it takes. It is also why Kaabist is blocked less often than a scraper that does not pace itself.

## What Kaabist has learned about a site

`GET /v1/sites/{hostname}` shows what the platform has learned about a site you use:

```json
{ "hostname": "shop.example.com", "object": "site_profile", "tier_required": "browser", "tier_confidence": 0.9, "known_obstacles_count": 2, "api_endpoints_count": 1, "last_blocked_at": "2026-09-30T08:12:00Z" }
```

The tier it needs and how confident Kaabist is, how many obstacles it has learned to dismiss there, how many JSON endpoints were found behind it, and when it last blocked a request. It is the answer to "why is this site slower and more expensive than that one". A hostname nothing is known about returns an empty profile, not an error. What your own proxy or model taught Kaabist stays in your organisation's profile and never reaches the shared one.

## When a request never leaves

Some requests are refused before a connection is opened:

| Reason | What happened | What the job sees |
|---|---|---|
| `ssrf_blocked` | The address the hostname resolves to is private, loopback, link-local or a cloud metadata endpoint, or the name ends in `.internal` or `.local`. This is not configurable: a program that could reach those could read Kaabist's own infrastructure. | fails with `ssrf_blocked` |
| `blocked_list` | The target is on Kaabist's blocklist, or on a blocklist your organisation configured. | the navigation fails |
| `rate_limited` | The site's allowance stayed spent for the whole twenty-second wait. | the navigation fails |
| `org_concurrency` | Your organisation already has as many connections open as it may. | the navigation fails |
| `host_concurrency` | The site already has as many connections open as Kaabist allows it, across all customers. | the navigation fails |
| `no_provider` | The requested pool has no provider configured at all. | the navigation fails |
| `upstream_error` | Two different proxy providers failed to open the connection. | the navigation fails |
| `byo_proxy_unreachable` | Your own proxy could not be reached and `fallback` is `none`. | fails with `byo_proxy_unreachable`, not retryable; `error.details.byo_id` names the proxy |

A redirect is re-checked at every hop, and the address a hostname resolves to is checked and then connected to directly, so a page that redirects to an internal address, or a name that resolves to a public address once and a private one the next time, is refused rather than followed.

## What Kaabist records

For each connection: the hostname, the bytes in each direction, how long it lasted, and which pool it used. Not the URL path, not the headers, and not the content: a secure connection is opaque to the gateway by design. Bytes are what proxy usage is billed on. See [Data retention](https://docs.kaab.ist/concepts/data-retention) for how long the rest of a job's data is kept.

## Worked example

**A redirect to an internal address.** A shop's link `https://shop.example.com/go/partner` answers with a `302` to `http://169.254.169.254/latest/meta-data/`, the cloud metadata address. A job that opens it reaches the shop, receives the redirect, and the gateway refuses the second hop: the job fails with `error.code: "ssrf_blocked"`. The same happens with a redirect to `127.0.0.1`. No setting changes this.

**A site that blocks datacenter addresses.** Some sites let only residential exits through. A job with the default proxy reaches it from a datacenter address and receives a challenge page; Kaabist recognises the challenge, moves to a residential exit and a stronger tier as your `max_tier` allows, and the job's `meta.proxy.pool` and `meta.escalations` show what it took. The residential traffic is billed at 0.5 credits a megabyte instead of 0.05.

**Your own proxy, with a fallback.** With a proxy registered as `byo_01J…`:

```json
{ "options": { "proxy": { "mode": "byo", "byo_id": "byo_01J…", "fallback": "platform" } } }
```

If your proxy is down when the job runs, the job continues through Kaabist's exits, its result carries `{ "code": "byo_fallback_used" }`, and that traffic is billed. With `"fallback": "none"` the same job fails with `byo_proxy_unreachable` and nothing leaves through an exit you did not choose.

## Pitfalls

- **A country nobody serves is replaced, not refused.** When the content depends on the country, test the job from that country before relying on it in a schedule.
- **Residential costs ten times datacenter.** Pin `pool: "datacenter"` only when you are sure the site accepts it; otherwise let Kaabist move up when it must.
- **`fallback: "platform"` is billed.** The fallback traffic goes through Kaabist's pools and is charged like any other.
- **Pacing is per site, not per job.** Splitting a crawl into more jobs does not make one site answer faster.
- **Development addresses are refused in production.** A program that works against `localhost` on your machine is refused the same address on Kaabist.

## Reference

- [Options](https://docs.kaab.ist/reference/dsl/options), including `proxy`
- [Get a site profile](https://docs.kaab.ist/reference/api/sites/get-site-profile)
- [Bring-your-own provider schema](https://docs.kaab.ist/reference/api/schemas/byo-provider) and [test a provider](https://docs.kaab.ist/reference/api/byo-providers/test-byo-provider)
- [Error codes](https://docs.kaab.ist/reference/codes#errors) and [warning codes](https://docs.kaab.ist/reference/codes#warnings)
- [Bring your own providers](https://docs.kaab.ist/guides/byo-providers) and [Security and data](https://docs.kaab.ist/guides/security-and-data)