# Lists and pagination

Extract lists of objects, follow "next" links across pages, and load infinite lists by scrolling, with bounds lint can check.

Most data worth automating comes as a list: products on a category page, rows in a report,
results spread over twelve pages or loaded as you scroll. This guide shows how to read a list of
objects, collect it across pages with `paginate`, and load infinite lists with `scroll` and
`loop_until`, always with a bound so a job cannot walk forever.

## When to use it

- One page, many items: a `many: true` field with `fields`.
- Items spread over numbered pages or a "next" link: a `paginate` step.
- Items that load as you scroll, or behind a "load more" button: `scroll … until`, or a
  `loop_until` that clicks the button.

If the page fetches its items from a JSON API, reading that API is often simpler and cheaper than
walking the rendered list; see [JSON and network locators](https://docs.kaab.ist/guides/json-and-network-locators).

## Lists of objects

`many: true` turns a field into a list. With `fields`, each match becomes an object, and every
subfield is located **inside** its own item, so the third price always belongs to the third
product:

```json
"extract": {
  "products": {
    "selector": ".product-card",
    "many": true,
    "min_items": 1,
    "fields": {
      "title": ".title",
      "price": { "selector": ".price", "type": "money", "locale": "tr", "currency": "TRY" },
      "url": { "selector": "a", "attr": "href", "type": "url" }
    }
  }
}
```

- Without `fields`, `many: true` gives a list of scalars (`"tags": { "selector": ".tag", "many": true }`).
- `fields` without `many` gives a single object, scoped to the first match.
- Inside an item, `selector: ":scope"` is the item itself, for a field that reads the item's own
  text or attribute.
- `min_items`, `max_items` and `unique` (`true`, or the name of a subfield such as `"sku"`)
  constrain the list.

### When the list is shorter than expected

A list with fewer items than `min_items` keeps the items it found and carries the warning
`list_short` on the list's path, for example `{ "code": "list_short", "field": "products" }`. A
job that would rather fail than return a short list sets `"on_invalid": "fail"` on that field.
`min_items` is your best defence against a page that quietly started showing fewer results.

Warnings about the items themselves are aggregated onto the list path with a count, so one bad
price in forty reads `{ "code": "null_marker", "field": "products[].price", "count": 1, "of": 40 }`
rather than forty separate entries. See
[validation and warnings](https://docs.kaab.ist/concepts/validation-and-warnings).

## Pages with a "next" link

A `paginate` step pairs a navigation with an `extract`. It runs the extract on the current page,
follows `next`, and repeats:

```json
{
  "paginate": {
    "next": { "css": "a[rel=next]" },
    "max_pages": 5,
    "stop_when": { "absent": "a[rel=next]" },
    "delay_ms": 0
  },
  "extract": { "products": { "selector": ".product-card", "many": true, "fields": { … } } }
}
```

- It stops at whichever comes first: `max_pages` pages, the `stop_when` condition being true on
  the page it has read, `next` not being found, or `next` no longer moving the page.
- Top-level `many: true` fields are **concatenated** across pages. Every other field is taken from
  the first page.
- `next` is an ordinary target, so it can carry `candidates` for a site that renders the link
  differently over time.

Two lint errors keep a paginate step honest:

- [E002](https://docs.kaab.ist/reference/dsl/lint/e002): a `paginate` without `max_pages` (and a `scroll … until` or a
  `loop_until` without `max`). Every loop in a program has a bound.
- [E010](https://docs.kaab.ist/reference/dsl/lint/e010): a `paginate` whose extract has no `many: true` field, which
  would read the first page's values and throw the rest away.

## Infinite lists

### Scrolling until a condition holds

`scroll` with `to` and `until` scrolls, waits `delay_ms`, checks the condition, and repeats up to
`max` times:

```json
{ "scroll": { "to": "bottom", "until": { "exists": "#done" }, "max": 10, "delay_ms": 100 } }
```

`max` is required whenever `until` is given ([E002](https://docs.kaab.ist/reference/dsl/lint/e002)).

### Repeating a block

`loop_until` runs a block of steps until a condition is true, checking the condition **before**
each pass, at most `max` times (1 to 100). Use it for a "load more" button, or to combine a scroll with
other steps:

```json
{
  "loop_until": { "exists": "#done" },
  "max": 10,
  "do": [ { "scroll": { "to": "bottom", "until": { "exists": "#done" }, "max": 10, "delay_ms": 100 } } ]
}
```

A `loop_until` that reaches `max` without its condition becoming true fails the job with
`loop_exhausted`, unless the step is `optional`.

### Virtualised lists

Some lists keep only the rows on screen in the page and recycle them as you scroll. Picture a list
of 1 000 rows that never has more than 20 in the page. Scrolling reaches the end,
but an `extract` afterwards reads only the rows in the page at that moment. For such a list, read
the data from the API that feeds it (see [JSON and network locators](https://docs.kaab.ist/guides/json-and-network-locators)),
or read what you need from what stays in the page, as this program does:

```json
[
  { "scroll": { "to": "bottom", "until": { "count": { "target": "#rows .row[data-index='999']", "eq": 1 } },
    "max": 60, "delay_ms": 50 } },
  { "extract": {
      "total": { "selector": "#total", "type": "integer" },
      "last_index": { "selector": "#rows .row:last-child", "attr": "data-index", "type": "integer" } } }
]
```

## Worked example

Take a shop at `https://shop.example.com` whose category list `/c/shoes?page=n` shows ten items a
page over twelve pages, each with a label and an amount in Turkish lira. This program collects
three pages:

### TypeScript

```ts
const definition = {
  kaab: "1",
  name,
  options: { max_tier: "browser" }, // following "next" clicks it, which takes a browser
  steps: [
    { goto: `${site}/list?page=1`, expect: { visible: "#items .item" } },
    {
      paginate: {
        next: { css: "a[rel=next]" },
        max_pages: 3,
        stop_when: { absent: "a[rel=next]" },
      },
      extract: {
        items: {
          selector: "#items .item",
          many: true,
          fields: {
            label: ".label",
            amount: { selector: ".amount", type: "money", locale: "tr", currency: "TRY" },
          },
        },
      },
    },
  ],
};
await kaab.programs.create({ name, definition, publish: true });

const started = await kaab.jobs.create({ program: name });
const job = await kaab.jobs.wait(started.id, { timeoutMs: 120_000 });
const items = (job.data?.items ?? []) as { label: string }[];
console.log(`${job.id}: ${items.length} items, ${items[0]?.label} … ${items.at(-1)?.label}`);
```

The job succeeds with 30 objects in `data.items`, `{ "label": …, "amount": … }` each, in page order.
Raise `max_pages` to 12 and it collects all 120; at 20 it still stops after page 12, because
`a[rel=next]` is absent on the last page.

Suppose the shop's search page is the infinite version: ten items at first, more from a cursor API
each time you reach the bottom, and a `#done` marker after the last batch. The `loop_until` and
`scroll` blocks above load it to the end; an `extract` with `"min_items": 50` on the items then
confirms the whole list arrived.

## Pitfalls

- **A `paginate` field that is not `many: true` comes from page one only.** That is by design (a
  page title, a total), but a list you forgot to mark `many` silently stops at one page's worth.
- **`stop_when` is checked on the page you have read.** Choose a condition that is true on the last
  page, such as the "next" link being absent, not one that is true on the page after it.
- **Long walks hit the step's time budget.** A `paginate` step may run for its `timeout_ms` once
  per page; a step that runs past that budget fails with `timeout`. Keep `max_pages` realistic and
  split very long walks across jobs with an input such as a start page.
- **Virtualised lists keep one screenful.** On a virtualised list, the DOM you extract from is the last
  screenful, not the whole list.
- **`unique` catches overlapping pages.** Sites that shift items between pages while you walk them
  produce duplicates; `"unique": "sku"` turns that into a visible warning.

## Reference

- [`paginate`](https://docs.kaab.ist/reference/dsl/actions/paginate), [`scroll`](https://docs.kaab.ist/reference/dsl/actions/scroll),
  [`loop_until`](https://docs.kaab.ist/reference/dsl/actions/loop-until)
- Fields: [`many`](https://docs.kaab.ist/reference/dsl/fields/many), [`fields`](https://docs.kaab.ist/reference/dsl/fields/fields),
  [`min-items`](https://docs.kaab.ist/reference/dsl/fields/min-items), [`unique`](https://docs.kaab.ist/reference/dsl/fields/unique),
  [`on_invalid`](https://docs.kaab.ist/reference/dsl/fields/on-invalid)
- Lint: [E002](https://docs.kaab.ist/reference/dsl/lint/e002), [E010](https://docs.kaab.ist/reference/dsl/lint/e010)
- [Conditions](https://docs.kaab.ist/reference/dsl/conditions/absent) such as `absent`, `exists` and
  [`count`](https://docs.kaab.ist/reference/dsl/conditions/count)
- [Warning codes](https://docs.kaab.ist/reference/codes#warnings), including `list_short`