Programs and versions

A program is a named, versioned JSON definition of what to do on a website and what data to return, and every job runs one exact version of one. Because a published version never changes, a job that ran last month is reproducible, and you decide when production moves to new behaviour.

#When to use it

Store a program as soon as a job is more than a one-off: when it signs in, clicks, runs on a schedule, or other people depend on its output. A stored program gets versions, aliases, diffs, healing proposals and the review queue. For a single public page read once, the url + extract shorthand is enough, and it compiles to the same kind of program.

#What a program is

{
  "kaab": "1",
  "name": "shop-product",
  "inputs": { "product_id": { "type": "string", "default": "1", "pattern": "^[0-9]+$" } },
  "options": { "max_tier": "http" },
  "steps": [
    { "goto": "https://shop.example.com/p/{{inputs.product_id}}" },
    { "extract": { "title": "h1", "price": { "selector": ".price", "type": "money", "locale": "tr", "currency": "TRY" } } }
  ]
}

kaab is the DSL version ("1"), name identifies the program in your organisation, inputs are typed values each job supplies, options set the tier, session, region and budgets, and steps run in order. The DSL reference lists every key.

#The one-step shorthand

A job body { "url": U, "extract": S, "options": O } compiles at request time to { "kaab": "1", "options": O, "steps": [ { "goto": U }, { "extract": S } ] }. There is no second, simpler engine: the shorthand is linted, run and billed exactly like a program you typed out, so moving from a shorthand job to a stored program changes nothing about how it runs.

#Three forms of the same program

#Versions

POST /v1/programs creates the program and its version 1, published unless you pass publish: false. Every later change is POST /v1/programs/{id}/versions with a new definition and optional notes. There is no endpoint that edits a published version, because a version is what a job's result is attributable to.

A version is either a draft or published:

• A draft can be replaced in place (PUT /v1/programs/{id}/versions/{n}) and discarded (DELETE), and is what the panel's builder autosaves into. It can be run by its pv_… id but is never what a bare name resolves to.
• POST /v1/programs/{id}/versions/{n}/publish publishes a draft. From then on the version is immutable, and jobs, schedules and aliases may point at it.

A version's identity is a hash of its canonical form. Two documents that differ only in shorthand, in the order of their keys, or in which defaults they wrote down are the same version.

#Aliases and references

The program field of a job or a schedule accepts five spellings:

latest is moved by publishing and by nothing else, so a bare name never resolves to a draft. Every other alias is yours: PUT /v1/programs/{id}/aliases/production with { "version": 2 } creates or moves it, and DELETE removes it. Alias names are lowercase letters, digits and hyphens, up to 32 characters, starting with a letter. Approving a healing proposal with promote: true publishes a new version and moves latest, and deliberately leaves production where it is.

A schedule stores the reference, not the version, and resolves it at every run: a schedule on shop-product@production follows the alias when you move it.

#Lint

Every program is linted when it is saved and when it is published. Errors block the write (422 program_invalid, with the findings in details.errors[]); warnings are returned and do not block. POST /v1/programs/lint returns both, plus the normalized document, without storing anything.

Errors are things that cannot work: an unknown key or wrong type, a loop or pagination with no bound, evaluate without its limits, a target with no locator or with two, a duplicate step id, a template naming an input or variable that does not exist, an invalid field name, a regular expression, glob or JSONPath that does not compile, paginate over a schema with no list, a side-effect step that asks to be retried or skipped, more than 200 steps or nesting deeper than three, a secret input used inside extract or rendered into a URL, and a download name that can produce no file name.

Warnings are about programs that work today and will break later:

• a goto or click with no expect, the step that fails silently;
• a locator with nothing but a structural path behind it, which the next redesign breaks;
• a fixed sleep where a condition would do;
• a side-effect step not followed by a checkpoint before the next side effect;
• a number or money field whose sample contains both . and , with neither locale nor decimal set;
• an evaluate, which the healer cannot see through;
• an await_input collected on a hosted page with nobody notified;
• a {{vars.x}} read after a block that sets x on only one branch, which renders an empty string.

Each code has its own page under lint codes.

#Diff and replay

GET /v1/programs/{id}/versions/{n}/diff?against=m returns a JSON Patch (RFC 6902) between two definitions, taken over the expanded form, so a change reads the way you wrote it. A healing proposal that relocates one field produces a diff with one operation in it.

POST /v1/programs/{id}/versions/{n}/replay runs a version's extraction against a page Kaabist already fetched, with no network:

{ "snapshot": "job_01J…:steps[1]" }

or { "snapshot": "ground_truth" } for the page the version's fields were confirmed against. It answers { data, raw, warnings } and costs no credits, which makes it the loop to use while you get a locator right. A version whose steps need a live page cannot be replayed, and says so.

#Deleting

DELETE /v1/programs/{id} (an admin key) is a soft delete. A program that still has schedules, active or paused, answers 409 program_has_schedules: delete them first, so nothing is left pointing at a program that no longer resolves.

#Worked example

This walks a shop's product page from a lint to a job that runs by alias. First, lint the definition shown above:

const definition = {
  kaab: "1",
  name,
  inputs: { product_id: { type: "string", default: "1", pattern: "^[0-9]+$" } },
  options: { max_tier: "http" },
  steps: [
    { goto: `${site}/p/{{inputs.product_id}}` },
    {
      extract: {
        title: "h1",
        price: { selector: ".price", type: "money", locale: "tr", currency: "TRY" },
      },
    },
  ],
};
const lint = await kaab.programs.lint({ definition });
if (lint.errors.length > 0) throw new Error(`lint: ${JSON.stringify(lint.errors)}`);
console.log(`lint: ${lint.errors.length} errors, ${lint.warnings.length} warnings`);

It has no errors, so create the program with version 1 published, then add version 2, which also reads the SKU, and publish it. latest now points at 2:

// Version 1, published: alias `latest` points at it.
const program = await kaab.programs.create({ name, definition, publish: true });

// Version 2 adds a field; publishing it moves `latest` to 2.
const v2 = await kaab.programs.versions.create(program.id, {
  definition: {
    ...definition,
    steps: [
      definition.steps[0],
      { extract: { ...definition.steps[1]?.extract, sku: { selector: ".sku" } } },
    ],
  },
  notes: "read the SKU too",
  publish: true,
});
console.log(`${program.name}: v${v2.version} is ${v2.status}`);

Point production at version 1. It stays there until you move it, whatever you publish:

// `production` moves only when you say so: here it stays on version 1.
const aliased = await kaab.programs.aliases.set(program.id, "production", { version: 1 });
console.log(`aliases: ${JSON.stringify(aliased.aliases)}`);

Run a job by alias. The envelope's program.version is 1, because production resolved to it, and data has a title and a price but no SKU:

const job = await kaab.jobs.create({
  program: `${name}@production`,
  inputs: { product_id: "2" },
  wait: 30,
});
if (job.status !== "succeeded") throw new Error(`job ${job.id} is ${job.status}`);
console.log(
  `${job.id} ran ${job.program?.name}@${job.program?.version}: ${JSON.stringify(job.data)}`,
);

To see what changed between the two versions, ask for the diff of version 2 against 1: a single add operation for the sku field of the second step's extract. To move production, set the alias to 2; jobs created after that run version 2, and jobs created before it keep the version they pinned.

#Pitfalls

• Expecting latest to be what production runs. latest moves on every publish, including a healed version you approved. Run production jobs and schedules by an alias you control, such as @production.
• Editing instead of versioning. A published version cannot change. Create a new version, test it by its pv_… id or a staging alias, then move production.
• Ignoring warnings. They do not block publishing, but W001 (no expect) and W002 (structural-only locators) are the usual cause of a program that returns nothing after a site redesign.
• Putting a secret in a URL. Lint refuses a secret input rendered into a goto URL or used in extract, because each of those writes it somewhere Kaabist keeps.

#Reference

• Create a program, create a version, publish a version
• Set an alias, diff two versions, replay a version
• Lint a program and the lint codes
• The program, program version and program definition objects
• kaab programs push