# Webhooks How Kaabist tells your systems that something happened: the events, the signature, retries, ordering and how to receive them safely. Kaabist sends an HTTP request to your server when something you care about happens: a job finished, a job needs a person, a program started drifting, credits are running low. Every delivery is signed, so you can prove it came from Kaabist and was not changed on the way, and your systems can react without polling. ## When to use it Use webhooks when your system should react to Kaabist rather than ask it: start processing when a job succeeds, page someone when one needs attention, act on a schedule's change, top up credits when they run low. Polling is the simpler choice when one script creates a job and waits for it: `Prefer: wait` on `POST /v1/jobs`, or an SDK's `jobs.wait`, needs no public endpoint at all. Both work together; a webhook is a notification, and the API stays the source of truth. A webhook is a notification, not a copy of your data. The body names what happened and where to read it (`data.url`); fetch the job, proposal or program from the API when you need the details. Job results, inputs, sessions and credentials are never in a webhook. ## Registering an endpoint `POST /v1/webhooks` with an `admin` key registers a URL and the events it wants: ### TypeScript ```ts const endpoint = await kaab.webhooks.create({ url, events: ["job.*"], description: "example" }); secret = endpoint.secret; // shown once: store it where the receiver can read it ``` ### Python ```python endpoint = kaab.webhooks.create(url=url, events=["job.*"], description="example") secret = endpoint.secret # shown once: store it where the receiver can read it ``` ### Go ```go body, err := kaab.Body[client.CreateWebhookJSONBody](map[string]any{"url": url, "events": []string{"job.*"}, "description": "example"}) if err != nil { log.Fatal(err) } endpoint, err := c.Webhooks.Create(ctx, body) if err != nil { log.Fatal(err) } defer func() { _ = c.Webhooks.Delete(ctx, endpoint.ID) }() mu.Lock() secret = endpoint.Secret // shown once: store it where the receiver can read it mu.Unlock() ``` The response carries `secret`: `whsec_` followed by 43 characters. **It is shown once.** Store it where your receiver can read it; Kaabist keeps only an encrypted copy. | Field | Meaning | |---|---| | `url` | `https://` only. It must resolve to a public address, every time it is called (see [where deliveries may go](https://docs.kaab.ist/guides/webhooks#where-deliveries-may-go)). | | `events` | Patterns: an exact type (`job.succeeded`), a group (`job.*`, every type starting `job.`), or `*` for everything. Unknown types are refused when the endpoint is saved. | | `mode` | `live`, `test` or `both` (the default): which keys' events the endpoint receives. A job created with an `sk_test_` key produces `mode: "test"` events. | | `format` | `standard` (the default): the signed event body described below. `discord` or `slack`: each event arrives as a chat message instead. | | `description` | Free text for your own bookkeeping. | `PATCH /v1/webhooks/{id}` changes any of them, `DELETE` removes the endpoint, and `POST /v1/webhooks/{id}/test` sends a `ping` event and waits up to ten seconds for your answer, which it returns as a delivery record. ### A job's own `webhook_url` A job can also name a one-off destination: `webhook_url` on `POST /v1/jobs` (or on a batch) receives that job's `job.*` events (or the batch's `batch.completed`) whether or not an endpoint matches. Those deliveries are always in the standard format and signed with your **organisation's** signing secret; get it with `POST /v1/org/webhook-secret/rotate` (send `{}`), which, like an endpoint rotation, keeps the previous secret valid for a day. ### Discord and Slack Discord's and Slack's incoming webhooks accept only their own message shape, and answer the standard event body with a `400`. Give such an endpoint the matching `format` and every event arrives as a message instead: ```json { "url": "https://discord.com/api/webhooks/…", "events": ["job.failed", "schedule.changed"], "format": "discord" } ``` The message says what happened ("Job failed", "Scheduled run found a change"), lists the event's `data` (the job, the program, the reason, the changed paths, …), and links to the job, schedule or program in the panel. It carries what the event carries and nothing more: a job's result is never in a webhook, so a price change shows as the changed path with a link to read the new value. Discord messages mention nobody, whatever a program is called. Deliveries are still signed and retried like any other. The panel picks the format by itself when you paste a Discord or Slack address. ## What a delivery looks like ```http POST /kaab HTTP/1.1 Content-Type: application/json User-Agent: Kaab-Webhooks/1 Kaab-Event-Id: evt_01JB8Q3Y2N6T4W9H5K7M1P0R3S Kaab-Event-Type: job.needs_attention Kaab-Signature: t=1758362400,v1=5257a869e7ecebeda32affa62cdca3fa51cad7e77a0e56ff536d0ce8e108d8bd {"id":"evt_01JB8Q3Y2N6T4W9H5K7M1P0R3S","type":"job.needs_attention","created_at":"2026-09-20T10:00:00.000Z","org":"org_01JB8Q2V…","mode":"live","data":{"job":"job_01JB8Q3X…","program":"acme-report@3","reason":"obstacle","url":"https://api.kaab.ist/v1/jobs/job_01JB8Q3X…"}} ``` | Key | Meaning | |---|---| | `id` | The event's id. The same event delivered twice has the same id; use it to deduplicate. | | `type` | One of the types below. | | `created_at` | When the event happened (not when it was delivered). | | `org` | Your organisation. | | `mode` | `live` or `test`. | | `data` | The event's fields, always including `url`: the API resource the event is about. | ## Events | Type | When | `data` | |---|---|---| | `job.queued` | A job was accepted, re-queued after a lease was lost, resumed, or escalated to a higher tier. | `job`, `program`, `url` | | `job.blocked` | A queued job is being held back: the organisation is out of credits. Sent once per blocked stretch. | `job`, `program`, `reason` (`insufficient_credits`), `url` | | `job.started` | A worker began running the job (its first run only). | `job`, `program`, `url` | | `job.awaiting_input` | The job parked at an `await_input` step and needs a value (`POST /jobs/{id}/input`). | `job`, `program`, `url` | | `job.needs_attention` | The job parked and needs a decision: an obstacle, drift, an unknown page, an expired session. | `job`, `program`, `reason` (the attention reason), `url` | | `job.succeeded` | The job finished and its result can be read. | `job`, `program`, `url` | | `job.failed` | The job ended without a result. | `job`, `program`, `reason` (the error code), `url` | | `job.canceled` | The job was canceled or aborted. | `job`, `program`, `reason` (`canceled`), `url` | | `batch.completed` | Every job of a batch has finished. | `batch`, `total`, `url` | | `proposal.created` | Kaabist proposed a fix (a relocated locator, a dismiss recipe) for review. | `proposal`, `program`, `job`, `kind`, `url` | | `proposal.decided` | Somebody approved or rejected a proposal. | `proposal`, `program`, `job`, `kind`, `decision`, `url` | | `schedule.run` | A scheduled job ended with nothing to report: the result is the same as last time (`changed: false`), or the job did not succeed (`status`). | `schedule`, `job`, `status`, `changed`?, `url` | | `schedule.changed` | A scheduled run's watched fields differ from the previous run's. | `schedule`, `job`, `changed_paths`, `changed_paths_truncated`?, `url`, `diff_url` | | `schedule.paused` | A schedule was paused: out of credits, or by hand. | `schedule`, `reason` (`credits` or `manual`), `url` | | `schedule.failing` | Five runs of a schedule in a row did not succeed; it keeps running. | `schedule`, `job`, `failure_streak`, `url` | | `session.expired` | A stored session stopped working. | `session`, `name`, `url` | | `program.drift_detected` | A field of a published version stopped filling, parsing or keeping its length, or its canary failed, before your next job notices. | `program`, `program_version_id`, `field`, `kind`, `detail`, `source`, `at`, `job` (canary only), `url` | | `credits.low` | Allowance consumption crossed 80 % or 95 %. Once per level per period. | `balance`, `level`, `url` | | `credits.exhausted` | Nothing more can be consumed. Once, until credits arrive. | `balance`, `url` | | `credits.recharged` | Credits arrived after exhaustion, or an auto-recharge completed. | `balance`, `url` | | `byo.unhealthy` | One of your own providers (proxy, model, captcha) started failing its health checks. | `byo`, `kind`, `status`, `detail`?, `url` | | `byo.recovered` | A provider that was failing or degraded passed two health checks in a row. | `byo`, `kind`, `status`, `url` | | `webhook.disabled` | Kaabist switched one of your endpoints off after a day of failed deliveries. | `webhook`, `reason` (`auto`), `failure_streak`, `failing_since`, `url` | | `ping` | You asked for a test delivery. Sent to that endpoint only; no pattern selects it. | `webhook`, `url` | `program` is `name@version` of the program the job ran, or `null` for a `url` + `extract` job. ## Verifying the signature Every delivery carries `Kaab-Signature: t=,v1=`. `v1` is the HMAC-SHA256, under your endpoint's secret (the whole `whsec_…` string), of the timestamp, a dot and the raw request body. For a day after you rotate a secret the header carries a second `v1`, made with the previous secret, so receivers that still have the old one keep working while you update them. Accept a delivery when **any** `v1` matches and the timestamp is within five minutes of now; refuse it otherwise. Two rules make or break this: - **Verify the raw body.** Read the bytes as they arrived and verify those. A body that was parsed and serialized again will not match. - **Compare in constant time.** Every example below does. The SDKs do both for you: the receiver below verifies with the SDK, answers at once and deduplicates on the event id. ### TypeScript ```ts const server = createServer((request, response) => { const chunks: Buffer[] = []; request.on("data", (chunk: Buffer) => chunks.push(chunk)); request.on("end", () => { try { // The raw bytes, before any JSON parser: the signature is over them. const event = kaab.webhooks.constructEvent( Buffer.concat(chunks), request.headers["kaab-signature"]?.toString(), secret, ); response.writeHead(204).end(); // answer fast; work afterwards if (seen.has(event.id)) return; seen.add(event.id); console.log(`verified ${event.type} ${event.id}`); resolveFirst(event.type); } catch (error) { response.writeHead(error instanceof KaabSignatureError ? 401 : 400).end(); console.log(`refused a delivery: ${(error as Error).message}`); } }); }); ``` ### Python ```python class Receiver(BaseHTTPRequestHandler): def do_POST(self) -> None: # noqa: N802 raw = self.rfile.read(int(self.headers.get("content-length", "0"))) # the bytes, unparsed try: event = webhooks.construct_event(raw, self.headers.get("kaab-signature"), secret) except KaabSignatureError as error: self.send_response(401) self.end_headers() print(f"refused a delivery: {error}") return self.send_response(204) # answer fast; work afterwards self.end_headers() if event["id"] not in seen: seen.add(event["id"]) print(f"verified {event['type']} {event['id']}") received.put(event["type"]) def log_message(self, *_: object) -> None: pass ``` ### Go ```go handler := http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { raw, err := io.ReadAll(r.Body) // the bytes, before any JSON decoding if err != nil { w.WriteHeader(http.StatusBadRequest) return } mu.Lock() key := secret mu.Unlock() event, err := webhooks.ConstructEvent(raw, r.Header.Get(webhooks.SignatureHeader), key) if err != nil { w.WriteHeader(http.StatusUnauthorized) log.Printf("refused a delivery: %v", err) return } w.WriteHeader(http.StatusNoContent) // answer fast; work afterwards mu.Lock() defer mu.Unlock() if !seen[event.ID] { seen[event.ID] = true fmt.Printf("verified %s %s\n", event.Type, event.ID) received <- event.Type } }) ``` ### Without an SDK These functions use the language's standard library only. They are the files `examples/webhooks/node/verify.ts`, `examples/webhooks/python/verify.py` and `examples/webhooks/go/verify.go`; Kaabist's test suite runs them against real deliveries on every change. Each directory also holds a small receiver that verifies, answers and deduplicates. ### TypeScript ```ts import { createHmac, timingSafeEqual } from "node:crypto"; /** Reject deliveries whose timestamp is more than five minutes from now. */ export const TOLERANCE_SECONDS = 300; /** * Verify a Kaab webhook. * * `header` is the `Kaab-Signature` request header, `rawBody` the request body exactly as it * arrived (read it before any JSON parser touches it), `secret` the endpoint's `whsec_…` secret. * The header is `t=,v1=`, with a second `v1` for a day after a secret * rotation; the delivery is genuine when any `v1` is the HMAC-SHA256 of `"."`. */ export function verifyKaabSignature( header: string | undefined, rawBody: string, secret: string, nowSeconds: number = Math.floor(Date.now() / 1000), ): boolean { if (header === undefined) return false; const [stamp, ...signatures] = header.split(","); if (stamp === undefined || !stamp.startsWith("t=")) return false; const timestamp = Number(stamp.slice(2)); if (!Number.isInteger(timestamp)) return false; if (Math.abs(nowSeconds - timestamp) > TOLERANCE_SECONDS) return false; const expected = createHmac("sha256", secret).update(`${timestamp}.${rawBody}`).digest(); return signatures.some((part) => { if (!part.startsWith("v1=")) return false; const given = Buffer.from(part.slice(3), "hex"); return given.length === expected.length && timingSafeEqual(given, expected); }); } ``` ### Python ```python import hashlib import hmac import json import sys import time TOLERANCE_SECONDS = 300 def verify_kaab_signature(header, raw_body, secret, now=None): """True when `header` (the Kaab-Signature header) signs `raw_body` under `secret`. `raw_body` is the request body exactly as it arrived, as bytes or str. The header is `t=,v1=`, with a second `v1` for a day after a secret rotation; the delivery is genuine when any `v1` is the HMAC-SHA256 of ".". """ if not header: return False parts = header.split(",") if not parts[0].startswith("t="): return False try: timestamp = int(parts[0][2:]) except ValueError: return False now = int(time.time()) if now is None else now if abs(now - timestamp) > TOLERANCE_SECONDS: return False body = raw_body.encode("utf-8") if isinstance(raw_body, str) else raw_body signed = str(timestamp).encode("ascii") + b"." + body expected = hmac.new(secret.encode("utf-8"), signed, hashlib.sha256).hexdigest() return any( part.startswith("v1=") and hmac.compare_digest(part[3:], expected) for part in parts[1:] ) if __name__ == "__main__": # Test harness: `{ "secret", "signature", "body", "now" }` as JSON on standard input. data = json.load(sys.stdin) ok = verify_kaab_signature(data["signature"], data["body"], data["secret"], data["now"]) print("valid" if ok else "invalid") ``` ### Go ```go import ( "crypto/hmac" "crypto/sha256" "encoding/hex" "strconv" "strings" "time" ) // ToleranceSeconds is how far a delivery's timestamp may be from now. const ToleranceSeconds = 300 // VerifyKaabSignature reports whether header (the Kaab-Signature request header) signs rawBody, // the request body exactly as it arrived, under secret, the endpoint's whsec_… secret. // // The header is t=,v1=, with a second v1 for a day after a secret rotation; // the delivery is genuine when any v1 is the HMAC-SHA256 of ".". func VerifyKaabSignature(header string, rawBody []byte, secret string, now time.Time) bool { parts := strings.Split(header, ",") if len(parts) < 2 || !strings.HasPrefix(parts[0], "t=") { return false } timestamp, err := strconv.ParseInt(parts[0][2:], 10, 64) if err != nil { return false } age := now.Unix() - timestamp if age > ToleranceSeconds || age < -ToleranceSeconds { return false } mac := hmac.New(sha256.New, []byte(secret)) mac.Write([]byte(strconv.FormatInt(timestamp, 10) + ".")) mac.Write(rawBody) expected := mac.Sum(nil) for _, part := range parts[1:] { if !strings.HasPrefix(part, "v1=") { continue } given, err := hex.DecodeString(part[3:]) if err == nil && hmac.Equal(given, expected) { return true } } return false } ``` ## Retries, ordering and failures **Success** is any `2xx` answer within **10 seconds**. Answer first and do the work afterwards; a receiver that takes longer is counted as failed and gets the event again. Redirects are not followed: a `3xx` is a failure. **Retries.** A failed delivery is tried again after 1 minute, 5 minutes, 30 minutes, 2 hours and 12 hours (six attempts over about 14½ hours) and is then marked `exhausted`. A delivery whose destination is refused is marked `failed` at once and not retried. Every attempt, its status code, its latency and the first kilobyte of your answer are in `GET /v1/webhooks/{id}/deliveries` (filter with `status` and `event_type`). **Ordering.** Kaabist delivers to each endpoint **in the order the events happened, one at a time.** A delivery that is failing therefore holds back the later events for *that endpoint* until it succeeds or exhausts its retries; other endpoints are not affected, however long that endpoint's backlog grows. An event whose announcement was lost and is delivered late joins the endpoint's line behind the delivery already at its head. Order across endpoints, and between an endpoint and a job's `webhook_url`, is not guaranteed. **Resending.** `POST /v1/webhooks/{id}/deliveries/{delivery_id}/resend` sends the same event again as a new delivery; the original record is left as it was. The event's `id` and body are the same, so a receiver that deduplicates treats it as a repeat. **Auto-disable.** An endpoint that has failed **100 deliveries in a row, over at least 24 hours,** is switched off: its `status` becomes `disabled` with `disabled_reason: "auto"`, a `webhook.disabled` event is recorded, and your organisation's owners and admins get an e-mail. Events keep being recorded while it is off. Fix the receiver, `PATCH` the endpoint back to `"status": "active"` and resend what it missed. ## Receiving safely - **Deduplicate on `id`.** Kaabist delivers at least once: a delivery that timed out on our side may have been processed on yours, and it will come again. Keep the ids you have handled (a unique column is enough) and answer `2xx` to a repeat without doing the work twice. - **Treat the body as a pointer.** Read the job, proposal or program from `data.url` with your API key before acting on it; the API is the source of truth, and it is where the data lives. - **Keep the secret out of source control**, and rotate it (`POST /v1/webhooks/{id}/rotate-secret`) if it leaks. The new secret is shown once; the old one keeps signing for a day. - **Use separate endpoints for test and live** (`mode`) if your receivers are separate. ## Testing locally `POST /v1/webhooks/{id}/test` answers with the delivery record (status, latency, the start of your answer), so you can see a misconfigured receiver without waiting for a real event. To receive real deliveries on a laptop, run a receiver locally and expose it through a tunnel, for example `cloudflared tunnel --url http://localhost:4400`, then register the tunnel's `https://` address as the endpoint's `url`. The example receivers listen on port 4400 by default (`KAAB_WEBHOOK_PORT`) and read the secret from `KAAB_WEBHOOK_SECRET`. ## Where deliveries may go Kaabist resolves an endpoint's hostname before every delivery and connects only to the address it checked. Every address the name resolves to must be public: private ranges (`10/8`, `172.16/12`, `192.168/16`, `100.64/10`), loopback, link-local, cloud metadata addresses, their IPv6 equivalents, `.internal` and `.local` names and numeric spellings like `2130706433` are refused. A delivery refused this way is recorded as `failed` with `error: "ssrf_blocked"` and is not retried. URLs with credentials in them (`https://user:pass@…`) are refused when the endpoint is saved. ## Worked example Receive a job's events from start to finish: 1. Register an endpoint for `["job.*"]` and keep the `secret` from the response. 2. `POST /v1/webhooks/{id}/test`. The answer is a delivery record with `event_type: "ping"` and `status: "succeeded"`, and your receiver logs a verified `ping`. 3. Run a job against a product page with `Prefer: wait=30`: ```json { "url": "https://shop.example.com/p/1", "extract": { "title": "h1" }, "options": { "max_tier": "http" } } ``` 4. Your receiver gets three deliveries for that job, in this order: `job.queued`, `job.started`, `job.succeeded`. Each carries `User-Agent: Kaab-Webhooks/1`, a `Kaab-Signature` that verifies with your secret (and fails with any other), and `data.url` pointing at `https://api.kaab.ist/v1/jobs/`. 5. `GET /v1/webhooks/{id}/deliveries?status=succeeded` lists the same deliveries with their status codes and latencies. ## Pitfalls - **Parsing before verifying.** A framework that parses JSON before your handler runs hands you a re-serialized body, and the signature no longer matches. Read the raw bytes first. - **Doing the work before answering.** Anything that pushes the answer past 10 seconds counts as a failure and brings the event back. Answer `2xx`, then process. - **Assuming exactly once.** Repeats happen, by design. Deduplicate on the event `id`. - **Assuming the payload holds the data.** It does not. Read the resource from `data.url`. - **One broken endpoint for everything.** A failing delivery holds back that endpoint's later events. Separate endpoints for separate consumers keep one outage from delaying the others. - **Clocks.** The five-minute window assumes your server's clock is right; keep it synchronised. ## Reference - API: [create a webhook](https://docs.kaab.ist/reference/api/webhooks/create-webhook), [send a test](https://docs.kaab.ist/reference/api/webhooks/test-webhook), [list deliveries](https://docs.kaab.ist/reference/api/webhooks/list-webhook-deliveries), [resend a delivery](https://docs.kaab.ist/reference/api/webhooks/resend-webhook-delivery), [rotate the secret](https://docs.kaab.ist/reference/api/webhooks/rotate-webhook-secret) - Schemas: [webhook](https://docs.kaab.ist/reference/api/schemas/webhook), [delivery](https://docs.kaab.ist/reference/api/schemas/webhook-delivery), [event](https://docs.kaab.ist/reference/api/schemas/webhook-event) - [Every event and its data](https://docs.kaab.ist/reference/events), [event types](https://docs.kaab.ist/reference/codes#events) - CLI: [`kaab webhooks create`](https://docs.kaab.ist/reference/cli/kaab-webhooks-create)