Docs
API 1.39
Status
Guides

Lists and pagination

Extract lists of objects, follow "next" links across pages, and load infinite lists by scrolling, with bounds lint can check.

Most data worth automating comes as a list: products on a category page, rows in a report, results spread over twelve pages or loaded as you scroll. This guide shows how to read a list of objects, collect it across pages with paginate, and load infinite lists with scroll and loop_until, always with a bound so a job cannot walk forever.

#When to use it

  • One page, many items: a many: true field with fields.
  • Items spread over numbered pages or a "next" link: a paginate step.
  • Items that load as you scroll, or behind a "load more" button: scroll … until, or a loop_until that clicks the button.

If the page fetches its items from a JSON API, reading that API is often simpler and cheaper than walking the rendered list; see JSON and network locators.

#Lists of objects

many: true turns a field into a list. With fields, each match becomes an object, and every subfield is located inside its own item, so the third price always belongs to the third product:

"extract": {
  "products": {
    "selector": ".product-card",
    "many": true,
    "min_items": 1,
    "fields": {
      "title": ".title",
      "price": { "selector": ".price", "type": "money", "locale": "tr", "currency": "TRY" },
      "url": { "selector": "a", "attr": "href", "type": "url" }
    }
  }
}
  • Without fields, many: true gives a list of scalars ("tags": { "selector": ".tag", "many": true }).
  • fields without many gives a single object, scoped to the first match.
  • Inside an item, selector: ":scope" is the item itself, for a field that reads the item's own text or attribute.
  • min_items, max_items and unique (true, or the name of a subfield such as "sku") constrain the list.

#When the list is shorter than expected

A list with fewer items than min_items keeps the items it found and carries the warning list_short on the list's path, for example { "code": "list_short", "field": "products" }. A job that would rather fail than return a short list sets "on_invalid": "fail" on that field. min_items is your best defence against a page that quietly started showing fewer results.

Warnings about the items themselves are aggregated onto the list path with a count, so one bad price in forty reads { "code": "null_marker", "field": "products[].price", "count": 1, "of": 40 } rather than forty separate entries. See validation and warnings.

A paginate step pairs a navigation with an extract. It runs the extract on the current page, follows next, and repeats:

{
  "paginate": {
    "next": { "css": "a[rel=next]" },
    "max_pages": 5,
    "stop_when": { "absent": "a[rel=next]" },
    "delay_ms": 0
  },
  "extract": { "products": { "selector": ".product-card", "many": true, "fields": { … } } }
}
  • It stops at whichever comes first: max_pages pages, the stop_when condition being true on the page it has read, next not being found, or next no longer moving the page.
  • Top-level many: true fields are concatenated across pages. Every other field is taken from the first page.
  • next is an ordinary target, so it can carry candidates for a site that renders the link differently over time.

Two lint errors keep a paginate step honest:

  • E002: a paginate without max_pages (and a scroll … until or a loop_until without max). Every loop in a program has a bound.
  • E010: a paginate whose extract has no many: true field, which would read the first page's values and throw the rest away.

#Infinite lists

#Scrolling until a condition holds

scroll with to and until scrolls, waits delay_ms, checks the condition, and repeats up to max times:

{ "scroll": { "to": "bottom", "until": { "exists": "#done" }, "max": 10, "delay_ms": 100 } }

max is required whenever until is given (E002).

#Repeating a block

loop_until runs a block of steps until a condition is true, checking the condition before each pass, at most max times (1 to 100). Use it for a "load more" button, or to combine a scroll with other steps:

{
  "loop_until": { "exists": "#done" },
  "max": 10,
  "do": [ { "scroll": { "to": "bottom", "until": { "exists": "#done" }, "max": 10, "delay_ms": 100 } } ]
}

A loop_until that reaches max without its condition becoming true fails the job with loop_exhausted, unless the step is optional.

#Virtualised lists

Some lists keep only the rows on screen in the page and recycle them as you scroll. Picture a list of 1 000 rows that never has more than 20 in the page. Scrolling reaches the end, but an extract afterwards reads only the rows in the page at that moment. For such a list, read the data from the API that feeds it (see JSON and network locators), or read what you need from what stays in the page, as this program does:

[
  { "scroll": { "to": "bottom", "until": { "count": { "target": "#rows .row[data-index='999']", "eq": 1 } },
    "max": 60, "delay_ms": 50 } },
  { "extract": {
      "total": { "selector": "#total", "type": "integer" },
      "last_index": { "selector": "#rows .row:last-child", "attr": "data-index", "type": "integer" } } }
]

#Worked example

Take a shop at https://shop.example.com whose category list /c/shoes?page=n shows ten items a page over twelve pages, each with a label and an amount in Turkish lira. This program collects three pages:

TypeScript
const definition = {
  kaab: "1",
  name,
  options: { max_tier: "browser" }, // following "next" clicks it, which takes a browser
  steps: [
    { goto: `${site}/list?page=1`, expect: { visible: "#items .item" } },
    {
      paginate: {
        next: { css: "a[rel=next]" },
        max_pages: 3,
        stop_when: { absent: "a[rel=next]" },
      },
      extract: {
        items: {
          selector: "#items .item",
          many: true,
          fields: {
            label: ".label",
            amount: { selector: ".amount", type: "money", locale: "tr", currency: "TRY" },
          },
        },
      },
    },
  ],
};
await kaab.programs.create({ name, definition, publish: true });

const started = await kaab.jobs.create({ program: name });
const job = await kaab.jobs.wait(started.id, { timeoutMs: 120_000 });
const items = (job.data?.items ?? []) as { label: string }[];
console.log(`${job.id}: ${items.length} items, ${items[0]?.label} … ${items.at(-1)?.label}`);

The job succeeds with 30 objects in data.items, { "label": …, "amount": … } each, in page order. Raise max_pages to 12 and it collects all 120; at 20 it still stops after page 12, because a[rel=next] is absent on the last page.

Suppose the shop's search page is the infinite version: ten items at first, more from a cursor API each time you reach the bottom, and a #done marker after the last batch. The loop_until and scroll blocks above load it to the end; an extract with "min_items": 50 on the items then confirms the whole list arrived.

#Pitfalls

  • A paginate field that is not many: true comes from page one only. That is by design (a page title, a total), but a list you forgot to mark many silently stops at one page's worth.
  • stop_when is checked on the page you have read. Choose a condition that is true on the last page, such as the "next" link being absent, not one that is true on the page after it.
  • Long walks hit the step's time budget. A paginate step may run for its timeout_ms once per page; a step that runs past that budget fails with timeout. Keep max_pages realistic and split very long walks across jobs with an input such as a start page.
  • Virtualised lists keep one screenful. On a virtualised list, the DOM you extract from is the last screenful, not the whole list.
  • unique catches overlapping pages. Sites that shift items between pages while you walk them produce duplicates; "unique": "sku" turns that into a visible warning.

#Reference

Was this page helpful?