DSL
Fields
Every option of an extract field.
An extract maps a field name to a selector string, or to a field object with these options. Validation and warnings explains types, parsing and what happens when a value does not validate.
| Option | Type | Description |
|---|---|---|
allow_suffix | boolean | number and integer fields: accept magnitude suffixes such as 1.2k and 1,5M. Defaults to false. |
attr | string | Reads an attribute instead of the text: an attribute name such as href, @html for the inner HTML or @outer for the outer HTML. Defaults to the normalized text content. |
base_url | string | url fields: the base a relative URL is resolved against. |
candidates | object[] | An ordered list of field locators, tried in order. A field with candidates carries no other locator key (lint E005). |
case_insensitive | boolean | enum fields: match values ignoring case and surrounding whitespace. Defaults to true. |
country | string | phone fields: the two-letter country used for numbers printed without a country code. Defaults to the one implied by locale. |
currency | string | money fields: the three-letter ISO currency code used when the page prints none. |
date_format | string or string[] | date and datetime fields: one ICU format pattern or a list of them. ISO-8601 is always accepted. |
decimal | string | The single-character decimal separator for numbers, overriding locale. |
false_values | string[] | boolean fields: strings read as false. The defaults include false, no, off, 0, hayır, nein, non and no. |
fields | fields | Child fields, each scoped to the matched element. With many: true the field is a list of objects, one per match; without it, a single object read from the first match. |
flags | string | Flags for regex: any of i, m, s and u. |
follow_redirects | boolean | url fields on a browser tier: a link that answers with a redirect is replaced by where it points, such as a search engine's click-through link. The link is asked once, from the job's exit, without following it. Defaults to false. |
frame | string[] | The chain of <iframe> selectors from the top document to the frame the field is in. |
json | string | A JSONPath evaluated over JSON embedded in the page, such as __NEXT_DATA__, window.__INITIAL_STATE__ and JSON script tags, in document order; the first non-empty result wins. An invalid path is lint E009. |
locale | string | A BCP-47 locale such as tr that sets the decimal, thousands and date defaults for parsing. Kaabist does not guess it; lint warns W005 when a numeric sample is ambiguous without it. |
many | boolean | When true, the field is a list with one entry per matched element: scalars, or objects when fields is present. |
max | number or string | The largest allowed value: a number, or an ISO-8601 string for date and datetime. A value above it warns out_of_range. |
max_items | integer | many fields: the most items allowed. |
max_length | integer | The longest allowed string, in characters. |
min | number or string | The smallest allowed value: a number, or an ISO-8601 string for date and datetime. A value below it warns out_of_range. |
min_items | integer | many fields: the fewest items required. |
min_length | integer | The shortest allowed string, in characters. |
name | string | The accessible name that qualifies a role locator. |
network | object | Reads the value from a captured JSON network response instead of the page. |
null_markers | string[] | Strings read as null. Defaults to "", -, the en and em dash characters, N/A, n/a, null and None. |
nullable | boolean | Whether the value may be null. Defaults to true. |
on_invalid | "null" | "raw" | "fail" | "drop" | What happens to a value that cannot be parsed or breaks a constraint: null (the default), raw keeps the unparsed string, fail fails the job with validation_failed, drop removes the item from its list (only inside a many field). |
pattern | string | A regular expression the parsed string must match. |
presence | boolean | boolean fields: true when the element exists, whatever it prints. Defaults to false. |
regex | string | A regular expression applied to the located element's text (or HTML, with source), or to the whole document when the field has no other locator. The first capture group is the value, or the whole match without one. |
required | boolean | Whether a missing value warns required_missing. Defaults to false. |
role | string | An ARIA role locator, usually with name. |
sample | object | A sample value recorded by the picker, kept as a permanent test of the field's parsing. |
selector | string | A CSS selector. Inside an item of a list, :scope is the item itself. |
source | "text" | "html" | What regex is applied to: text (the default) or html. |
strip_fragment | boolean | url fields: drop the #fragment while normalizing. Defaults to false. |
testid | string | The value of the element's test id attribute. |
text | string | Locates the element by the text it shows. |
thousands | string | The thousands separator for numbers, at most one character (empty for none), overriding locale. |
timezone | string | datetime fields: the IANA time zone of times printed without an offset. |
transforms | "trim" | "lowercase" | "uppercase" | "strip_html" | "first" | "last" or object[] | Transforms applied in order to the raw string before it is parsed. |
true_values | string[] | boolean fields: strings read as true. The defaults include true, yes, on, 1, evet, ja, oui and sí. |
type | "string" | "text" | "number" | "integer" | "money" | "boolean" | "date" | "datetime" | "url" | "email" | "phone" | "enum" | "list" | "object" | The value's type: string (the default), text, number, integer, money, boolean, date, datetime, url, email, phone, enum, list or object. list and object are implied by many and fields. A number, integer or money reads the first number the text holds: a leading sign counts, a dash or plus further in ends the number, so the range 10-20 reads 10 and 1.299-1.499 TL (tr) reads 1299. |
unique | boolean or string | Lists only: true requires the items to be distinct; a field name requires that field to be distinct across a list of objects. |
values | string[] | enum fields: the allowed values, at least one. |
xpath | string | An XPath expression locator. |