# Validation and warnings How Kaabist turns page text into typed values, what it does when a value is missing or unreadable, and every warning a job can carry. Every extracted field goes through the same pipeline: find it, keep the raw string, parse it with the configuration your program gives, check its constraints, and apply its policy. When something is off, you get a typed `null` and a warning that says why, rather than a guess that looks like data. ## When to use it Read this page when you write the `extract` part of a program, when a result has warnings you need to act on, or when you decide how strict a job should be. The defaults favour keeping the job useful: one unreadable price out of forty is content, and the other thirty-nine still arrive. A program that would rather fail than return anything doubtful sets `policies.strict` or per-field `on_invalid: "fail"`. ## The pipeline ``` locate → raw → null markers → transforms → parse → constraints → policy ``` 1. **Locate**: the field's locator finds an element, a JSON value or a captured response. See [Locators](https://docs.kaab.ist/concepts/locators). 2. **Raw**: the element's normalised text (or the attribute named by `attr`) is kept as the raw string. 3. **Null markers**: a raw string that is a null marker becomes `null`. 4. **Transforms**: pure, bounded string operations (`trim`, `replace`, `split`, `slice` and the rest), in order. 5. **Parse**: the string becomes the field's type, using the field's parse configuration. 6. **Constraints**: `min`, `max`, `pattern` and the rest are checked against the parsed value. 7. **Policy**: a failure at steps 5 or 6 is handled by the field's `on_invalid`. Cross-field `$asserts` run after every field is read. ### The raw string is always kept Whatever a field ends up as, the string it was read from is kept. A price that came out `null` because the page printed an em dash (U+2014) is a page behaving normally; the same `null` from a page that printed `1.337,40 TL` is a parser configured wrongly, and only the raw string tells the two apart. A job returns `raw` for every field that failed or came out null. Set `policies.include_raw: true` to get it for every field. ## Types | `type` | Produces | |---|---| | `string` (default) | the text, as printed | | `text` | multi-line text with whitespace normalised | | `number`, `integer` | a number, read with the field's locale; `allow_suffix: true` also reads `1.2k`, `1,5M` | | `money` | `{ "amount": 1337.4, "currency": "TRY" }` | | `boolean` | `true` / `false` from `true_values` / `false_values`; `presence: true` reads whether the element exists | | `date` | `YYYY-MM-DD` | | `datetime` | an ISO-8601 string with an explicit offset | | `url` | an absolute URL, resolved against the page (or `base_url`); `javascript:` is refused | | `email` | an address | | `phone` | E.164, with the region from `country` or the locale | | `enum` | one of `values` (case-insensitive by default) | | `list`, `object` | implied by `many: true` and `fields` | ## Parsing numbers, money and dates ### Kaabist does not guess your locale `1.299` is one thousand two hundred and ninety-nine in Turkey and one and 299 thousandths in the United States, and no amount of cleverness makes one of those readings correct. So a numeric field is read with the `locale` (or the explicit `decimal` and `thousands`) the program gives it, and with `en` when it gives none. The builder proposes a configuration from the value you clicked on, and shows you the runner-up reading when the sample cannot decide for itself; publishing the program is where the choice becomes yours. Lint warns (`W005`) about a `number` or `money` field with no locale whose sample contains both `.` and `,`. Two places do read a value without being told how, because the value says so itself: - a number that came out of the page's own embedded JSON is already in JSON's format, and is read that way whatever `locale` says; - a price whose printed shape settles the question (`1.337,40` has its decimal separator last in every locale) is read from the shape when the field named no locale. ### Locales These are the locales the parser knows. A tag it does not know falls back to its language (`de-AT` reads as `de`), and an unknown language to `en`. A space in the thousands column means any space: an ordinary space, a no-break space and a narrow no-break space are the same mark. | `locale` | Decimal | Thousands | Month names | Default phone region | |---|---|---|---|---| | `en` | `.` | `,` | yes | none | | `en-us` | `.` | `,` | yes | US | | `en-gb` | `.` | `,` | yes | GB | | `en-in` | `.` | `,` | yes | IN | | `tr` | `,` | `.` | yes | TR | | `de` | `,` | `.` | yes | DE | | `de-ch` | `.` | `'` | yes | CH | | `fr` | `,` | space | yes | FR | | `es` | `,` | `.` | yes | ES | | `it` | `,` | `.` | yes | IT | | `pt` | `,` | `.` | yes | PT | | `pt-br` | `,` | `.` | yes | BR | | `nl` | `,` | `.` | yes | NL | | `pl` | `,` | space | no | PL | | `ru` | `,` | space | yes | RU | | `sv` | `,` | space | no | SE | | `da` | `,` | `.` | no | DK | | `fi` | `,` | space | no | FI | | `nb`, `no` | `,` | space | no | NO | | `cs` | `,` | space | no | CZ | | `hu` | `,` | space | no | HU | | `ro` | `,` | `.` | no | RO | | `ar` | `.` | `,` | no | none | | `ja` | `.` | `,` | no | JP | | `zh` | `.` | `,` | no | CN | | `ko` | `.` | `,` | no | KR | "Month names" means a `date_format` with `MMMM` or `MMM` can read the locale's month names (`12 Ekim 2026` in `tr`). ### Money The currency is read from the string when the page prints one (`TL`, `₺`, `TRY`, `€`, `$` and the other common symbols and codes) and falls back to the field's `currency` when it does not. The amount is read with, in order: the field's `locale`; the currency's own locale, for currencies that have exactly one (TRY reads as `tr`, BRL as `pt-br`, CHF as `de-ch`, JPY as `ja`, and so on); the printed shape, when it is unambiguous; and finally `en`. ### Dates A `date` or `datetime` field accepts ISO-8601 always, then each pattern in `date_format` (a string or a list, in ICU tokens such as `dd.MM.yyyy`) in order. A value printed without an offset is read in the field's `timezone`, `UTC` by default. Relative dates such as "2 days ago" are not parsed: the moment a page was rendered is not the moment a job re-reads it, and a date that changes on replay is worse than no date. ## Null markers A raw string that matches a null marker becomes `null`, with the warning `null_marker` (or `field_empty` when the string was empty). The default markers are `""`, `-`, the em dash (U+2014), the en dash (U+2013), `N/A`, `n/a`, `null` and `None`; set `null_markers` on a field to replace the list. A marker is not a failure: it is a page saying "no value", and the field's policy does not apply. ## Constraints | Key | Applies to | Fails as | |---|---|---| | `required` (default `false`) | any field | `required_missing` | | `nullable` (default `true`) | any field; `false` means a `null` is not acceptable | `required_missing` | | `min`, `max` | numbers, money, and `date`/`datetime` (as ISO-8601 strings) | `out_of_range` | | `min_length`, `max_length` | strings | `out_of_range` | | `pattern` | strings (a regular expression) | `out_of_range` | | `values` | `enum` | `out_of_range` | | `min_items` | lists | `list_short` | | `max_items` | lists | `out_of_range` | | `unique` | lists of scalars, or `unique: "field"` for a list of objects | `out_of_range` | `required` and `nullable: false` are not governed by `on_invalid`: a required field that is missing or null **fails the job** with `validation_failed`, unless the step is `optional`. ## What happens when a field fails A field that does not parse, or breaks a constraint, is handled by its `on_invalid`: | `on_invalid` | The field | The job | |---|---|---| | `null` (default) | `null`, status `invalid` | continues | | `raw` | keeps the unparsed string | continues | | `fail` | `null` | fails with `validation_failed` | | `drop` (list items only) | the item is removed from its list, status `dropped` | continues | Every case adds the matching warning (`parse_failed`, `out_of_range` or `list_short`). Lint refuses `drop` outside a list. ### Field statuses | Status | What happened | Warning | |---|---|---| | `ok` | found and parsed | none | | `null` | the page printed a null marker | `null_marker`, or `field_empty` for an empty string | | `invalid` | found but not parsable, or broke a constraint | `parse_failed`, `out_of_range`, `list_short` | | `missing` | no locator matched | `field_empty`, or `required_missing` when the field is required | | `dropped` | removed from its list by `on_invalid: "drop"` | the warning that caused it | ### `policies.strict` Set `"policies": { "strict": true }` on a job or a program to turn **any** extraction warning into a failure: the job fails with `validation_failed`, and `error.details` lists the fields and warnings that caused it. Use it when a downstream system would rather have nothing than something it has to check. A job that is not strict and failed after at least one `extract` ran still returns what it extracted. ## Cross-field checks `$asserts` compares two fields, or a field against a literal, with `<`, `<=`, `==`, `!=`, `>=` or `>`: ```json { "$asserts": [{ "left": "price", "op": "<=", "right": "list_price" }] } ``` A failing assert adds `out_of_range` on the left-hand field, under that field's own policy, or fails the job when it is strict. An assert whose side has no value is skipped, not failed. ## Warnings Warnings are a closed set; a job never carries a code outside this list. Each has a `code` and, where it applies, `field`, `raw`, `step`, `message`, `detail`, `count` and `of`. | Code | When it appears | |---|---| | `null_marker` | the page printed a null marker for a field | | `field_empty` | a field's element printed nothing, or an optional field's locator matched nothing; on a step, a template referenced a value that was missing | | `parse_failed` | a field's text could not be parsed as its type | | `out_of_range` | a parsed value broke `min`, `max`, `min_length`, `max_length`, `pattern`, `values`, `max_items` or `unique`, or a `$asserts` comparison failed | | `required_missing` | a `required` or `nullable: false` field had no value; the job fails unless the step is optional | | `list_short` | a list had fewer items than `min_items` | | `fill_rate_drop` | across the program version's recent runs, this field has stopped filling as it used to; a fix is probably waiting in the review queue (see [How healing works](https://docs.kaab.ist/concepts/self-healing)) | | `escalated` | the job moved to a stronger tier to finish; `message` says why (see [Tiers and escalation](https://docs.kaab.ist/concepts/tiers-and-escalation)) | | `healed_locator_used` | a step's locator no longer matched and a healed one carried this run only | | `obstacle_dismissed` | something covering the page was closed before the step ran; `detail` names how (see [Obstacles](https://docs.kaab.ist/concepts/obstacles)) | | `step_skipped` | an `optional` step, or one with `on_error: "skip"`, failed and was skipped; `message` says what failed | | `cache_hit` | reserved for a result served from the cache; check `meta.cache`, which is `"hit"` for such a result | | `byo_fallback_used` | your own proxy failed and `fallback: "platform"` sent the traffic through Kaabist's (see [Egress and proxies](https://docs.kaab.ist/concepts/egress-and-proxies)) | `fill_rate_drop` is the one warning that is not about *this* job: the data in this result may look fine. It carries the finding in `detail`, for example `products[].price: fill rate 97% → 20% over the last 340 samples`, and it clears when the field recovers or a healed version is published. ### List warnings are aggregated A warning that applies to items of a list is reported once for the list, with how many items it covered, instead of once per item: ```json { "code": "null_marker", "field": "products[].price", "count": 3, "of": 40 } ``` ## Worked example A shop's catalogue at `https://shop.example.com` prints Turkish prices (`1.337,40 TL`) on forty product cards, and the product pages of the app at `https://app.example.com` print `-` for a price that JavaScript fills in later. A list field over the catalogue: ```json { "kaab": "1", "options": { "max_tier": "http" }, "steps": [ { "goto": "https://shop.example.com/?page=1" }, { "extract": { "products": { "selector": ".product-card", "many": true, "min_items": 1, "fields": { "title": ".title", "price": { "selector": ".price", "type": "money", "locale": "tr", "currency": "TRY", "min": 0, "null_markers": ["-", "N/A"], "sample": { "raw": "1.337,40 TL", "value": 1337.4 } }, "url": { "selector": "a", "attr": "href", "type": "url" } } } } } ], "policies": { "include_raw": true } } ``` Each card's price comes back as `{ "amount": 1337.4, "currency": "TRY" }`, each `url` absolute against the page, and `raw` holds `1.337,40 TL` beside it because `include_raw` is on. Now change one thing at a time and watch the warnings: - **Remove `"locale": "tr"`.** The result does not change: the currency is TRY, and TRY reads as `tr`. Remove `"currency"` too, and the printed `TL` still names it. The shape `1.337,40` would settle it on its own. - **Read the same field with `"locale": "en"` on a value like `1.299 TL`.** It reads as one point two nine nine, which is why the locale is yours to set: `1.299` has no shape that decides. - **Read the same `price` field from `https://app.example.com/p/1` with `max_tier: "http"`.** The server printed `-` (JavaScript fills the real price in later), so `price` is `null` with a `null_marker` warning whose `raw` is `-`, and the job succeeds. Make the field `"required": true` and the job fails with `validation_failed` instead. [Tiers and escalation](https://docs.kaab.ist/concepts/tiers-and-escalation) shows how an `expect` on the price lets the job move to a browser and read the filled value instead. - **Add `"max": 100`.** Every price breaks the constraint: each is `null`, and one aggregated `out_of_range` warning covers the list. With `"on_invalid": "drop"`, the items are removed instead; with `"on_invalid": "fail"`, the job fails. To try a change like this without fetching the page again, replay the new version against a stored snapshot; see [Snapshots](https://docs.kaab.ist/concepts/snapshots#replaying-a-program-against-a-snapshot). ## Pitfalls - **Set a locale on every number and money field.** Without one, the parser reads `en`, which is wrong for most of the world's prices. - **A `null` with no warning is not possible.** If a field is `null`, the warnings say whether the page printed nothing, printed a marker, or printed something your configuration could not read. Read `raw` before changing the locator. - **`required` fails the job, `on_invalid` does not by default.** Use `required` for the fields a result is useless without, and leave the rest nullable. - **`strict` counts every extraction warning.** A single `null_marker` on an optional field fails a strict job; give such fields their own `null_markers` or accept them in your code instead. - **Relative dates do not parse.** Read the absolute date from an attribute (`attr: "datetime"` on a `