Testing: replay, dry-run and test mode

Kaabist gives you several ways to try a program before it matters: test-mode keys run real jobs for free, an estimate tells you what a job will cost without running it, lint checks a definition before you publish it, and a replay runs a new version against a page Kaabist already stored. Used together, they let you change a program with confidence and without surprises on your bill.

#When to use it

Testing does not replace watching production: a site can change the day after your tests pass. That is what canaries and drift detection are for; see drift and canaries.

#Test-mode keys

An API key's prefix decides the mode of every job it creates: sk_live_… runs live jobs, sk_test_… runs test jobs, and a request cannot override it.

A test job is a real job: it fetches the site, runs the browser and returns the same envelope. What differs:

• it reserves and spends no credits;
• instead, each plan allows a number of test jobs per UTC day (50 on Free, 200 on Developer, 1 000 on Team; see plans). Past it, creating one answers 429 test_quota_exceeded with Retry-After until midnight UTC;
• its result is kept for 24 hours;
• its events carry mode: "test", and a webhook endpoint registered with mode: "live" does not receive them;
• a schedule created with a test key runs test jobs.

// The key decides the mode: an sk_test_ key runs test jobs, which the live data never sees.
const probe = await kaab.jobs.create({
  url: `${site}/p/1`,
  extract: { title: "h1" },
  wait: 30,
});
console.log(`${probe.id} ran in ${probe.mode} mode`);

Keep test and live keys apart in your configuration, the same way you keep separate databases: CI and staging get sk_test_, production gets sk_live_.

#Estimating a job's cost

POST /v1/jobs/estimate takes exactly the body of POST /v1/jobs and answers without running anything:

{ "credits_min": …, "credits_max": …, "tier_expected": "http", "cache": "possible" }

tier_expected is where the job is expected to run, and cache says whether a cached result could serve it. The range is deliberately generous: the upper end includes a safety margin. An estimate needs only a read key, is free and never enqueues a job.

const estimate = await kaab.jobs.estimate({
  url: `${site}/p/1`,
  extract: { title: "h1", price: { selector: ".price", type: "money" } },
});
console.log(
  `between ${estimate.credits_min} and ${estimate.credits_max} credits at ${estimate.tier_expected} tier`,
);

#Linting a definition

POST /v1/programs/lint checks a definition without saving it, and answers errors, warnings and the normalized program. Errors block publishing: creating or publishing a version with lint errors is refused with 422 program_invalid, carrying the same findings. Warnings are advice, such as a goto without an expect (W001).

const definition = {
  kaab: "1",
  name,
  inputs: { product_id: { type: "string", default: "1", pattern: "^[0-9]+$" } },
  options: { max_tier: "http" },
  steps: [
    { goto: `${site}/p/{{inputs.product_id}}` },
    {
      extract: {
        title: "h1",
        price: { selector: ".price", type: "money", locale: "tr", currency: "TRY" },
      },
    },
  ],
};
const lint = await kaab.programs.lint({ definition });
if (lint.errors.length > 0) throw new Error(`lint: ${JSON.stringify(lint.errors)}`);
console.log(`lint: ${lint.errors.length} errors, ${lint.warnings.length} warnings`);

Run lint in CI on every program you keep in your repository; the kaab programs lint command does the same from a shell.

#Dry runs

"dry_run": true on POST /v1/jobs runs the program until its first side_effect step and stops there: the job is succeeded, meta.dry_run is true, and nothing marked as a side effect happened. Use it to test a program that submits an order or sends a message. See side-effect safety.

#Replaying a version against a stored page

Every step of a job stores a snapshot of its page. A replay runs a program version's extraction against one of those snapshots, without fetching anything, and returns { data, raw, warnings }:

{ "snapshot": "job_…:steps[1]" }

The snapshot is named by the job and the step path; "ground_truth" instead uses the page the version's fields were confirmed against in the builder. A replay costs 0 credits, needs a run key, and works on a draft, so you can check a change before you publish it:

// Run version 1 once: every step stores a snapshot of the page.
const run = await kaab.jobs.create({ program: `${name}@1`, wait: 30 });

// Version 2 reads one more field. Try it on the stored page: no fetch, no credits.
const v2 = await kaab.programs.versions.create(program.id, {
  publish: false, // a draft until the replay looks right
  definition: {
    kaab: "1",
    name,
    options: { max_tier: "http" },
    steps: [
      { goto: `${site}/p/3` },
      {
        extract: {
          title: "h1",
          price: { selector: ".price", type: "money" },
          sku: { selector: ".sku", regex: "SKU:\\s*([A-Z0-9-]+)" },
        },
      },
    ],
  },
});
const replay = await kaab.programs.versions.replay(program.id, v2.version, {
  snapshot: `${run.id}:steps[1]`,
});
console.log(`v${v2.version} on ${run.id}'s page: ${JSON.stringify(replay.data)}`);

A replay answers "does the new version read that page correctly". It does not navigate, log in or click, so it tests the extract, not the steps before it.

#Worked example

Change a program safely, from definition to replay, on a shop's product pages at https://shop.example.com:

1. Lint the definition: no errors, so it may be published.
2. Estimate a job on /p/1: a credit range and tier_expected: "http".
3. Run it with a test key: the job's mode is test, and your credit balance does not move.
4. Publish version 1 and run it once on /p/3. Every step stores a snapshot.
5. Create version 2 as a draft that also reads the SKU, with "sku": { "selector": ".sku", "regex": "SKU:\\s*([A-Z0-9-]+)" }.
6. Replay version 2 on job_…:steps[1], the extract step of step 4's job. data.sku holds the SKU, read from the stored page with no fetch and no credits. Publish version 2 when it looks right.

#Pitfalls

• A test key in production. Test results disappear after 24 hours and are capped per day. Check mode on a job if numbers look wrong.
• Treating an estimate as a price. The job's settled cost is what it used, usually below credits_max; an escalation to a higher tier is why the range exists.
• Replays only see one page. A version whose change is in the steps (a new click, another page) cannot be checked by a replay; run it as a test job.
• Snapshots expire. Snapshots follow your plan's retention. Replay against a recent job, or against ground_truth.
• Dry runs stop at the first side effect. Everything after it is untested; test that part with an sk_test_ key against an account where the side effect is harmless.

#Reference

• API: estimate a job, create a job (dry_run), lint a program, replay a version
• Lint result schema, lint rules
• Modes, plans (test_mode_daily_jobs)
• Concepts: keys and modes, snapshots, limits