Validation and warnings

Every extracted field goes through the same pipeline: find it, keep the raw string, parse it with the configuration your program gives, check its constraints, and apply its policy. When something is off, you get a typed null and a warning that says why, rather than a guess that looks like data.

#When to use it

Read this page when you write the extract part of a program, when a result has warnings you need to act on, or when you decide how strict a job should be. The defaults favour keeping the job useful: one unreadable price out of forty is content, and the other thirty-nine still arrive. A program that would rather fail than return anything doubtful sets policies.strict or per-field on_invalid: "fail".

#The pipeline

locate → raw → null markers → transforms → parse → constraints → policy

1. Locate: the field's locator finds an element, a JSON value or a captured response. See Locators.
2. Raw: the element's normalised text (or the attribute named by attr) is kept as the raw string.
3. Null markers: a raw string that is a null marker becomes null.
4. Transforms: pure, bounded string operations (trim, replace, split, slice and the rest), in order.
5. Parse: the string becomes the field's type, using the field's parse configuration.
6. Constraints: min, max, pattern and the rest are checked against the parsed value.
7. Policy: a failure at steps 5 or 6 is handled by the field's on_invalid.

Cross-field $asserts run after every field is read.

#The raw string is always kept

Whatever a field ends up as, the string it was read from is kept. A price that came out null because the page printed an em dash (U+2014) is a page behaving normally; the same null from a page that printed 1.337,40 TL is a parser configured wrongly, and only the raw string tells the two apart.

A job returns raw for every field that failed or came out null. Set policies.include_raw: true to get it for every field.

#Types

#Parsing numbers, money and dates

#Kaabist does not guess your locale

1.299 is one thousand two hundred and ninety-nine in Turkey and one and 299 thousandths in the United States, and no amount of cleverness makes one of those readings correct. So a numeric field is read with the locale (or the explicit decimal and thousands) the program gives it, and with en when it gives none. The builder proposes a configuration from the value you clicked on, and shows you the runner-up reading when the sample cannot decide for itself; publishing the program is where the choice becomes yours. Lint warns (W005) about a number or money field with no locale whose sample contains both . and ,.

Two places do read a value without being told how, because the value says so itself:

• a number that came out of the page's own embedded JSON is already in JSON's format, and is read that way whatever locale says;
• a price whose printed shape settles the question (1.337,40 has its decimal separator last in every locale) is read from the shape when the field named no locale.

#Locales

These are the locales the parser knows. A tag it does not know falls back to its language (de-AT reads as de), and an unknown language to en. A space in the thousands column means any space: an ordinary space, a no-break space and a narrow no-break space are the same mark.

"Month names" means a date_format with MMMM or MMM can read the locale's month names (12 Ekim 2026 in tr).

#Money

The currency is read from the string when the page prints one (TL, ₺, TRY, €, $ and the other common symbols and codes) and falls back to the field's currency when it does not. The amount is read with, in order: the field's locale; the currency's own locale, for currencies that have exactly one (TRY reads as tr, BRL as pt-br, CHF as de-ch, JPY as ja, and so on); the printed shape, when it is unambiguous; and finally en.

#Dates

A date or datetime field accepts ISO-8601 always, then each pattern in date_format (a string or a list, in ICU tokens such as dd.MM.yyyy) in order. A value printed without an offset is read in the field's timezone, UTC by default. Relative dates such as "2 days ago" are not parsed: the moment a page was rendered is not the moment a job re-reads it, and a date that changes on replay is worse than no date.

#Null markers

A raw string that matches a null marker becomes null, with the warning null_marker (or field_empty when the string was empty). The default markers are "", -, the em dash (U+2014), the en dash (U+2013), N/A, n/a, null and None; set null_markers on a field to replace the list. A marker is not a failure: it is a page saying "no value", and the field's policy does not apply.

#Constraints

required and nullable: false are not governed by on_invalid: a required field that is missing or null fails the job with validation_failed, unless the step is optional.

#What happens when a field fails

A field that does not parse, or breaks a constraint, is handled by its on_invalid:

Every case adds the matching warning (parse_failed, out_of_range or list_short). Lint refuses drop outside a list.

#Field statuses

#policies.strict

Set "policies": { "strict": true } on a job or a program to turn any extraction warning into a failure: the job fails with validation_failed, and error.details lists the fields and warnings that caused it. Use it when a downstream system would rather have nothing than something it has to check. A job that is not strict and failed after at least one extract ran still returns what it extracted.

#Cross-field checks

$asserts compares two fields, or a field against a literal, with <, <=, ==, !=, >= or >:

{ "$asserts": [{ "left": "price", "op": "<=", "right": "list_price" }] }

A failing assert adds out_of_range on the left-hand field, under that field's own policy, or fails the job when it is strict. An assert whose side has no value is skipped, not failed.

#Warnings

Warnings are a closed set; a job never carries a code outside this list. Each has a code and, where it applies, field, raw, step, message, detail, count and of.

fill_rate_drop is the one warning that is not about this job: the data in this result may look fine. It carries the finding in detail, for example products[].price: fill rate 97% → 20% over the last 340 samples, and it clears when the field recovers or a healed version is published.

#List warnings are aggregated

A warning that applies to items of a list is reported once for the list, with how many items it covered, instead of once per item:

{ "code": "null_marker", "field": "products[].price", "count": 3, "of": 40 }

#Worked example

A shop's catalogue at https://shop.example.com prints Turkish prices (1.337,40 TL) on forty product cards, and the product pages of the app at https://app.example.com print - for a price that JavaScript fills in later. A list field over the catalogue:

{
  "kaab": "1",
  "options": { "max_tier": "http" },
  "steps": [
    { "goto": "https://shop.example.com/?page=1" },
    {
      "extract": {
        "products": {
          "selector": ".product-card",
          "many": true,
          "min_items": 1,
          "fields": {
            "title": ".title",
            "price": {
              "selector": ".price",
              "type": "money",
              "locale": "tr",
              "currency": "TRY",
              "min": 0,
              "null_markers": ["-", "N/A"],
              "sample": { "raw": "1.337,40 TL", "value": 1337.4 }
            },
            "url": { "selector": "a", "attr": "href", "type": "url" }
          }
        }
      }
    }
  ],
  "policies": { "include_raw": true }
}

Each card's price comes back as { "amount": 1337.4, "currency": "TRY" }, each url absolute against the page, and raw holds 1.337,40 TL beside it because include_raw is on.

Now change one thing at a time and watch the warnings:

• Remove "locale": "tr". The result does not change: the currency is TRY, and TRY reads as tr. Remove "currency" too, and the printed TL still names it. The shape 1.337,40 would settle it on its own.
• Read the same field with "locale": "en" on a value like 1.299 TL. It reads as one point two nine nine, which is why the locale is yours to set: 1.299 has no shape that decides.
• Read the same price field from https://app.example.com/p/1 with max_tier: "http". The server printed - (JavaScript fills the real price in later), so price is null with a null_marker warning whose raw is -, and the job succeeds. Make the field "required": true and the job fails with validation_failed instead. Tiers and escalation shows how an expect on the price lets the job move to a browser and read the filled value instead.
• Add "max": 100. Every price breaks the constraint: each is null, and one aggregated out_of_range warning covers the list. With "on_invalid": "drop", the items are removed instead; with "on_invalid": "fail", the job fails.

To try a change like this without fetching the page again, replay the new version against a stored snapshot; see Snapshots.

#Pitfalls

• Set a locale on every number and money field. Without one, the parser reads en, which is wrong for most of the world's prices.
• A null with no warning is not possible. If a field is null, the warnings say whether the page printed nothing, printed a marker, or printed something your configuration could not read. Read raw before changing the locator.
• required fails the job, on_invalid does not by default. Use required for the fields a result is useless without, and leave the rest nullable.
• strict counts every extraction warning. A single null_marker on an optional field fails a strict job; give such fields their own null_markers or accept them in your code instead.
• Relative dates do not parse. Read the absolute date from an attribute (attr: "datetime" on a <time> element) when the page offers one.

#Reference

• Extract schema and the field options: type, locale, currency, null-markers, required, on-invalid, transforms, sample
• Assertions and policies
• Warning codes and error codes
• Job schema and extract schema