# Testing: replay, dry-run and test mode

Test programs without spending credits or touching live data: test keys, cost estimates, lint, dry runs and replays against stored pages.

Kaabist gives you several ways to try a program before it matters: test-mode keys run real jobs for
free, an estimate tells you what a job will cost without running it, lint checks a definition
before you publish it, and a replay runs a new version against a page Kaabist already stored. Used
together, they let you change a program with confidence and without surprises on your bill.

## When to use it

| You want to know | Use |
|---|---|
| Is this definition valid? | `POST /v1/programs/lint` |
| What will this job cost, and at which tier? | `POST /v1/jobs/estimate` |
| Does it work end to end, without spending credits? | a job created with an `sk_test_` key |
| Does it work without performing its side effects? | `dry_run: true` on `POST /v1/jobs` |
| Does my new version read the page the old one read? | `POST /v1/programs/{id}/versions/{n}/replay` |
| Does it keep working after I stop watching? | canaries and drift detection |

Testing does not replace watching production: a site can change the day after your tests pass.
That is what canaries and drift detection are for; see [drift and canaries](https://docs.kaab.ist/guides/drift-and-canaries).

## Test-mode keys

An API key's prefix decides the mode of every job it creates: `sk_live_…` runs live jobs,
`sk_test_…` runs test jobs, and a request cannot override it.

A test job is a **real** job: it fetches the site, runs the browser and returns the same envelope.
What differs:

- it reserves and spends **no credits**;
- instead, each plan allows a number of test jobs per UTC day (50 on Free, 200 on Developer, 1 000
  on Team; see [plans](https://docs.kaab.ist/reference/plans)). Past it, creating one answers `429 test_quota_exceeded`
  with `Retry-After` until midnight UTC;
- its result is kept for **24 hours**;
- its events carry `mode: "test"`, and a webhook endpoint registered with `mode: "live"` does not
  receive them;
- a schedule created with a test key runs test jobs.

### TypeScript

```ts
// The key decides the mode: an sk_test_ key runs test jobs, which the live data never sees.
const probe = await kaab.jobs.create({
  url: `${site}/p/1`,
  extract: { title: "h1" },
  wait: 30,
});
console.log(`${probe.id} ran in ${probe.mode} mode`);
```

Keep test and live keys apart in your configuration, the same way you keep separate databases:
CI and staging get `sk_test_`, production gets `sk_live_`.

## Estimating a job's cost

`POST /v1/jobs/estimate` takes exactly the body of `POST /v1/jobs` and answers without running
anything:

```json
{ "credits_min": …, "credits_max": …, "tier_expected": "http", "cache": "possible" }
```

`tier_expected` is where the job is expected to run, and `cache` says whether a cached result could
serve it. The range is deliberately generous: the upper end includes a safety margin.
An estimate needs only a `read` key, is free and never
enqueues a job.

### TypeScript

```ts
const estimate = await kaab.jobs.estimate({
  url: `${site}/p/1`,
  extract: { title: "h1", price: { selector: ".price", type: "money" } },
});
console.log(
  `between ${estimate.credits_min} and ${estimate.credits_max} credits at ${estimate.tier_expected} tier`,
);
```

## Linting a definition

`POST /v1/programs/lint` checks a definition without saving it, and answers `errors`, `warnings` and
the `normalized` program. Errors block publishing: creating or publishing a version with lint
errors is refused with `422 program_invalid`, carrying the same findings. Warnings are advice, such
as a `goto` without an `expect` ([W001](https://docs.kaab.ist/reference/dsl/lint/w001)).

### TypeScript

```ts
const definition = {
  kaab: "1",
  name,
  inputs: { product_id: { type: "string", default: "1", pattern: "^[0-9]+$" } },
  options: { max_tier: "http" },
  steps: [
    { goto: `${site}/p/{{inputs.product_id}}` },
    {
      extract: {
        title: "h1",
        price: { selector: ".price", type: "money", locale: "tr", currency: "TRY" },
      },
    },
  ],
};
const lint = await kaab.programs.lint({ definition });
if (lint.errors.length > 0) throw new Error(`lint: ${JSON.stringify(lint.errors)}`);
console.log(`lint: ${lint.errors.length} errors, ${lint.warnings.length} warnings`);
```

Run lint in CI on every program you keep in your repository; the [`kaab programs lint`](https://docs.kaab.ist/reference/cli/kaab-programs-lint)
command does the same from a shell.

## Dry runs

`"dry_run": true` on `POST /v1/jobs` runs the program until its first `side_effect` step and stops
there: the job is `succeeded`, `meta.dry_run` is `true`, and nothing marked as a side effect
happened. Use it to test a program that submits an order or sends a message. See
[side-effect safety](https://docs.kaab.ist/guides/side-effect-safety).

## Replaying a version against a stored page

Every step of a job stores a snapshot of its page. A replay runs a program version's extraction
against one of those snapshots, **without fetching anything**, and returns `{ data, raw, warnings }`:

```json
{ "snapshot": "job_…:steps[1]" }
```

The snapshot is named by the job and the step path; `"ground_truth"` instead uses the page the
version's fields were confirmed against in the builder. A replay costs 0 credits, needs a `run` key,
and works on a draft, so you can check a change before you publish it:

### TypeScript

```ts
// Run version 1 once: every step stores a snapshot of the page.
const run = await kaab.jobs.create({ program: `${name}@1`, wait: 30 });

// Version 2 reads one more field. Try it on the stored page: no fetch, no credits.
const v2 = await kaab.programs.versions.create(program.id, {
  publish: false, // a draft until the replay looks right
  definition: {
    kaab: "1",
    name,
    options: { max_tier: "http" },
    steps: [
      { goto: `${site}/p/3` },
      {
        extract: {
          title: "h1",
          price: { selector: ".price", type: "money" },
          sku: { selector: ".sku", regex: "SKU:\\s*([A-Z0-9-]+)" },
        },
      },
    ],
  },
});
const replay = await kaab.programs.versions.replay(program.id, v2.version, {
  snapshot: `${run.id}:steps[1]`,
});
console.log(`v${v2.version} on ${run.id}'s page: ${JSON.stringify(replay.data)}`);
```

A replay answers "does the new version read *that* page correctly". It does not navigate, log in
or click, so it tests the extract, not the steps before it.

## Worked example

Change a program safely, from definition to replay, on a shop's product pages at
`https://shop.example.com`:

1. **Lint** the definition: no errors, so it may be published.
2. **Estimate** a job on `/p/1`: a credit range and `tier_expected: "http"`.
3. **Run it with a test key**: the job's `mode` is `test`, and your credit balance does not move.
4. **Publish version 1** and run it once on `/p/3`. Every step stores a snapshot.
5. **Create version 2 as a draft** that also reads the SKU, with
   `"sku": { "selector": ".sku", "regex": "SKU:\\s*([A-Z0-9-]+)" }`.
6. **Replay version 2** on `job_…:steps[1]`, the extract step of step 4's job. `data.sku` holds the
   SKU, read from the stored page with no fetch and no credits. Publish version 2 when it looks
   right.

## Pitfalls

- **A test key in production.** Test results disappear after 24 hours and are capped per day.
  Check `mode` on a job if numbers look wrong.
- **Treating an estimate as a price.** The job's settled cost is what it used, usually below
  `credits_max`; an escalation to a higher tier is why the range exists.
- **Replays only see one page.** A version whose change is in the steps (a new click, another
  page) cannot be checked by a replay; run it as a test job.
- **Snapshots expire.** Snapshots follow your plan's retention. Replay against a recent job, or
  against `ground_truth`.
- **Dry runs stop at the first side effect.** Everything after it is untested; test that part with
  an `sk_test_` key against an account where the side effect is harmless.

## Reference

- API: [estimate a job](https://docs.kaab.ist/reference/api/jobs/estimate-job), [create a job](https://docs.kaab.ist/reference/api/jobs/create-job)
  (`dry_run`), [lint a program](https://docs.kaab.ist/reference/api/programs/lint-program),
  [replay a version](https://docs.kaab.ist/reference/api/programs/replay-program-version)
- [Lint result schema](https://docs.kaab.ist/reference/api/schemas/lint-result), [lint rules](https://docs.kaab.ist/reference/dsl/lint)
- [Modes](https://docs.kaab.ist/reference/codes#modes), [plans](https://docs.kaab.ist/reference/plans) (`test_mode_daily_jobs`)
- Concepts: [keys and modes](https://docs.kaab.ist/getting-started/keys-and-modes), [snapshots](https://docs.kaab.ist/concepts/snapshots),
  [limits](https://docs.kaab.ist/concepts/limits)