# Drift detection and canaries
How Kaabist notices that a program has quietly started returning worse data, and how it finds out before your next job does.
Drift detection watches the data your published programs return, and canaries run your programs on
a schedule so that a changed site is noticed before your own jobs depend on it. Together they turn
the quiet failure, a field that slowly stops filling while every job still succeeds, into a signal,
a webhook and a healing proposal waiting for review.
## When to use it
Rely on drift detection for every published program that runs regularly: it needs no setup and
works from your own traffic. Configure a canary for a program you run rarely (weekly, monthly, on
demand) or one that signs in, because statistics need recent runs to compare and a rare program
would otherwise tell you it broke a week late.
Neither is a test suite. To check a program against known pages before you publish it, use
[testing](https://docs.kaab.ist/guides/testing); to review and approve the fixes either one produces, see
[self-healing](https://docs.kaab.ist/concepts/self-healing).
## Statistics, from your own traffic
Every job records, per field, how many items it was looked for in, how many produced a value, how
many parsed, how long the lists were and what the numbers looked like. Those counts are folded into
an hourly rolling picture of each **published program version**.
Once an hour, each version with enough recent traffic (at least 20 jobs in the last seven days, not
counting canaries) is compared against itself:
| Window | What it is |
|---|---|
| baseline | the trailing 7 days, **ending 6 hours ago** |
| window | the last 6 hours |
The gap between them is the point. A baseline that included the present would drift toward whatever
broke, and a field that has been empty for three days would look normal.
A finding is opened when:
| Signal | Fires when |
|---|---|
| `fill_rate_drop` | the fill rate fell by 30 points **and** to below 90 % of its baseline |
| `parse_rate_drop` | the same, for values your parser could read |
| `list_len_drop` | the median list length halved |
| `value_shift` | the median value moved by more than a factor of ten (with at least 30 numbers on each side) |
Both rate rules need **two** conditions, because either alone is wrong in a way you would see the
same week: an absolute drop alone fires on every noisy sparse field, and a relative drop alone fires
when 4 % becomes 2 %. A window with fewer than ten samples produces nothing at all: a quiet night
is not evidence.
## Canaries, before you have any traffic at all
A **canary** is Kaabist running your program itself, on a schedule, against the site.
```json
{
"options": {
"canary": { "enabled": true, "cron": "0 */6 * * *", "url": "https://shop.example.com/p/1" }
}
}
```
- **On by default** for a program used in the last seven days that has somewhere to point: a
`canary.url`, or a program that is only `goto` and `extract` and so has no side effects. Set
`enabled: false` to opt out, `true` to opt in for anything else.
- **`url`** is passed to the run as the input named `url`, so a program whose first step is
`{ "goto": "{{inputs.url}}" }` probes that page. A program whose `goto` is a fixed address keeps
probing that address. `inputs` supplies any other inputs the run needs.
- **Every six hours** by default, in UTC.
- **Half price.** A canary bills compute at 50 %, and nothing else changes: same tier, same program,
same ledger. Canary runs are excluded from your usage figures and from the baselines above, so
they never flatter or distort your own numbers.
- **Never when you are out of credits**, and never on a program that already has an open drift
signal.
The canary can be set in the program's `options` or on the program itself (`PATCH /v1/programs/{id}`
with `canary`), which overrides the definition key by key. Credentials for a canary of a logged-in
flow are stored encrypted on the program, not in its definition.
## What happens when something is found
Four things, at once:
1. A **drift signal** opens, carrying both windows, so you can read `97% → 20% over 340 samples`
rather than "drift detected". One signal per field per kind, however long it stays broken.
2. The **healer is asked to look**, at the actual pages the failing runs captured. By the time your
next job runs, a proposal is usually already in your review queue
([self-healing](https://docs.kaab.ist/concepts/self-healing)).
3. A **`program.drift_detected` webhook** fires, once. It also fires when a canary run fails.
4. Jobs of that version start carrying a **`fill_rate_drop` warning** with the detail, so a client
that checks warnings finds out without subscribing to anything.
## When it goes away
A signal closes when the field is healthy for two consecutive windows: two rather than one, because
a program that runs four times a night can produce a quiet hour that looks like a recovery. It also
closes when you publish a healed version or dismiss it, and it expires after thirty days so that a
review queue reflects this month's problems.
## Worked example
A shop at `https://shop.example.com` ships a redesign that moves every locator and keeps every
value: the price moves from `.price` to `span[data-col=amount]`. This walkthrough follows a program
through that redesign.
### 1. Publish a program with a canary
The program reads `h1` and `.price` from product `{{inputs.product_id}}` of the shop, one locator
per field. Turn its canary on, probing product 1 every six hours:
### TypeScript
```ts
// Probe the published program every six hours; a failing probe sends program.drift_detected.
const updated = await kaab.programs.update(program.id, {
canary: { enabled: true, cron: "0 */6 * * *", inputs: { product_id: "1" } },
});
console.log(`${updated.name}: canary ${JSON.stringify(updated.canary)}`);
```
When the version is created, Kaabist stores the page it was confirmed on and a fingerprint of each
matched element: the ground truth the healer works from. Before the redesign, every run fills
`price`.
### 2. The site ships a redesign
The shop's product page changes its markup. Before, the price was
`1.337,40 TL`; after, it is
`1.337,40 TL`. The value is the same, but the program's `.price`
locator now points at nothing.
### 3. The next run notices
`.price` no longer matches. The healer compares the page with the ground truth and finds the
element that holds the same kind of value in the same place: `span[data-col=amount]`. When it is
confident enough, the run uses the new locator for this job only and carries a
`healed_locator_used` warning; when it is not, the job parks `needs_attention` with the reason
`drift`. A canary run that fails also fires `program.drift_detected`.
### 4. Review the proposal
Either way, a `relocate` proposal for `price` is pending, with the old and new locators, the values
each one read and the healer's confidence. List what is pending, then approve the proposal with
`promote: true` (or reject it with a reason):
### TypeScript
```ts
// What healing found and is waiting on a person: the old and new locator, and how sure it is.
let pending = 0;
for await (const proposal of kaab.proposals.list({ status: "pending", limit: 25 })) {
pending++;
console.log(
`${proposal.id} ${proposal.kind} ${proposal.target} confidence ${proposal.confidence} (${proposal.source})`,
);
// Then: kaab.proposals.approve(proposal.id, { promote: true }) or .reject(id, { reason })
if (pending >= 25) break;
}
console.log(`${pending} pending proposal(s)`);
```
Promoting publishes a new program version with the new locator and moves the alias `latest` to it
(`production` moves only when you move it). The next run on the redesigned page succeeds without
warnings.
## Pitfalls
- **A canary with nowhere to go.** A program with steps other than `goto` and `extract` gets no
canary by default, because Kaabist will not repeat actions on its own. Give it a `canary.url` and
`enabled: true` only if a run has no side effects.
- **Expecting a signal from a handful of runs.** Fewer than 20 jobs a week, or fewer than ten in the
last six hours, produce no statistics. Use a canary for rare programs.
- **Dismissing a signal you have not fixed.** It closes, and the next finding opens a new one only
when the numbers cross the thresholds again.
- **Ignoring `healed_locator_used`.** A healed job succeeded with a locator that was not approved.
Review the proposal; until you promote it, every run heals again.
- **Promoting and expecting `production` to follow.** Promotion moves `latest`; move `production`
yourself when you are ready.
## Reference
- [Program options: `canary`](https://docs.kaab.ist/reference/dsl/options) and [update a program](https://docs.kaab.ist/reference/api/programs/update-program)
- [List proposals](https://docs.kaab.ist/reference/api/proposals/list-proposals),
[approve a proposal](https://docs.kaab.ist/reference/api/proposals/approve-proposal)
- [Events: `program.drift_detected`](https://docs.kaab.ist/reference/events) and [warnings](https://docs.kaab.ist/reference/codes#warnings)
- [Tiers and escalation](https://docs.kaab.ist/concepts/tiers-and-escalation), the other half of "why did this job not
return what I expected"