Drift detection and canaries

Drift detection watches the data your published programs return, and canaries run your programs on a schedule so that a changed site is noticed before your own jobs depend on it. Together they turn the quiet failure, a field that slowly stops filling while every job still succeeds, into a signal, a webhook and a healing proposal waiting for review.

#When to use it

Rely on drift detection for every published program that runs regularly: it needs no setup and works from your own traffic. Configure a canary for a program you run rarely (weekly, monthly, on demand) or one that signs in, because statistics need recent runs to compare and a rare program would otherwise tell you it broke a week late.

Neither is a test suite. To check a program against known pages before you publish it, use testing; to review and approve the fixes either one produces, see self-healing.

#Statistics, from your own traffic

Every job records, per field, how many items it was looked for in, how many produced a value, how many parsed, how long the lists were and what the numbers looked like. Those counts are folded into an hourly rolling picture of each published program version.

Once an hour, each version with enough recent traffic (at least 20 jobs in the last seven days, not counting canaries) is compared against itself:

The gap between them is the point. A baseline that included the present would drift toward whatever broke, and a field that has been empty for three days would look normal.

A finding is opened when:

Both rate rules need two conditions, because either alone is wrong in a way you would see the same week: an absolute drop alone fires on every noisy sparse field, and a relative drop alone fires when 4 % becomes 2 %. A window with fewer than ten samples produces nothing at all: a quiet night is not evidence.

#Canaries, before you have any traffic at all

A canary is Kaabist running your program itself, on a schedule, against the site.

{
  "options": {
    "canary": { "enabled": true, "cron": "0 */6 * * *", "url": "https://shop.example.com/p/1" }
  }
}

• On by default for a program used in the last seven days that has somewhere to point: a canary.url, or a program that is only goto and extract and so has no side effects. Set enabled: false to opt out, true to opt in for anything else.
• url is passed to the run as the input named url, so a program whose first step is { "goto": "{{inputs.url}}" } probes that page. A program whose goto is a fixed address keeps probing that address. inputs supplies any other inputs the run needs.
• Every six hours by default, in UTC.
• Half price. A canary bills compute at 50 %, and nothing else changes: same tier, same program, same ledger. Canary runs are excluded from your usage figures and from the baselines above, so they never flatter or distort your own numbers.
• Never when you are out of credits, and never on a program that already has an open drift signal.

The canary can be set in the program's options or on the program itself (PATCH /v1/programs/{id} with canary), which overrides the definition key by key. Credentials for a canary of a logged-in flow are stored encrypted on the program, not in its definition.

#What happens when something is found

Four things, at once:

1. A drift signal opens, carrying both windows, so you can read 97% → 20% over 340 samples rather than "drift detected". One signal per field per kind, however long it stays broken.
2. The healer is asked to look, at the actual pages the failing runs captured. By the time your next job runs, a proposal is usually already in your review queue (self-healing).
3. A program.drift_detected webhook fires, once. It also fires when a canary run fails.
4. Jobs of that version start carrying a fill_rate_drop warning with the detail, so a client that checks warnings finds out without subscribing to anything.

#When it goes away

A signal closes when the field is healthy for two consecutive windows: two rather than one, because a program that runs four times a night can produce a quiet hour that looks like a recovery. It also closes when you publish a healed version or dismiss it, and it expires after thirty days so that a review queue reflects this month's problems.

#Worked example

A shop at https://shop.example.com ships a redesign that moves every locator and keeps every value: the price moves from .price to span[data-col=amount]. This walkthrough follows a program through that redesign.

#1. Publish a program with a canary

The program reads h1 and .price from product {{inputs.product_id}} of the shop, one locator per field. Turn its canary on, probing product 1 every six hours:

// Probe the published program every six hours; a failing probe sends program.drift_detected.
const updated = await kaab.programs.update(program.id, {
  canary: { enabled: true, cron: "0 */6 * * *", inputs: { product_id: "1" } },
});
console.log(`${updated.name}: canary ${JSON.stringify(updated.canary)}`);

When the version is created, Kaabist stores the page it was confirmed on and a fingerprint of each matched element: the ground truth the healer works from. Before the redesign, every run fills price.

#2. The site ships a redesign

The shop's product page changes its markup. Before, the price was <span class="price">1.337,40 TL</span>; after, it is <span data-col="amount">1.337,40 TL</span>. The value is the same, but the program's .price locator now points at nothing.

#3. The next run notices

.price no longer matches. The healer compares the page with the ground truth and finds the element that holds the same kind of value in the same place: span[data-col=amount]. When it is confident enough, the run uses the new locator for this job only and carries a healed_locator_used warning; when it is not, the job parks needs_attention with the reason drift. A canary run that fails also fires program.drift_detected.

#4. Review the proposal

Either way, a relocate proposal for price is pending, with the old and new locators, the values each one read and the healer's confidence. List what is pending, then approve the proposal with promote: true (or reject it with a reason):

// What healing found and is waiting on a person: the old and new locator, and how sure it is.
let pending = 0;
for await (const proposal of kaab.proposals.list({ status: "pending", limit: 25 })) {
  pending++;
  console.log(
    `${proposal.id} ${proposal.kind} ${proposal.target} confidence ${proposal.confidence} (${proposal.source})`,
  );
  // Then: kaab.proposals.approve(proposal.id, { promote: true }) or .reject(id, { reason })
  if (pending >= 25) break;
}
console.log(`${pending} pending proposal(s)`);

Promoting publishes a new program version with the new locator and moves the alias latest to it (production moves only when you move it). The next run on the redesigned page succeeds without warnings.

#Pitfalls

• A canary with nowhere to go. A program with steps other than goto and extract gets no canary by default, because Kaabist will not repeat actions on its own. Give it a canary.url and enabled: true only if a run has no side effects.
• Expecting a signal from a handful of runs. Fewer than 20 jobs a week, or fewer than ten in the last six hours, produce no statistics. Use a canary for rare programs.
• Dismissing a signal you have not fixed. It closes, and the next finding opens a new one only when the numbers cross the thresholds again.
• Ignoring healed_locator_used. A healed job succeeded with a locator that was not approved. Review the proposal; until you promote it, every run heals again.
• Promoting and expecting production to follow. Promotion moves latest; move production yourself when you are ready.

#Reference

• Program options: canary and update a program
• List proposals, approve a proposal
• Events: program.drift_detected and warnings
• Tiers and escalation, the other half of "why did this job not return what I expected"