# JSON and network locators

Read the JSON a page embeds or fetches instead of the HTML drawn from it, capture API responses, and let a discovered API move a job to the http tier.

Many pages are drawn from JSON: a server-rendered page embeds it in a script tag, a single-page app
fetches it from an API after loading. Reading that JSON directly is more precise than reading the
HTML made from it, survives redesigns, and can let a job run at the cheaper http tier without a
browser.

## When to use it

- **The data is in the page source as JSON** (`__NEXT_DATA__`, `window.__INITIAL_STATE__`,
  `application/ld+json`): use a `json` locator. It works at every tier.
- **The page fetches the data from an API while it loads**: use a `network` locator, or a
  `capture_response` step when you need the response in a variable for later steps.
- **You do not know where the data comes from**: run the program at the browser tier with ordinary
  CSS locators, and let API discovery propose the JSON locators for you.

Stay with CSS locators when the values only exist in the rendered page (a price computed in the
browser, text assembled from several responses), or when the API needs a token the page negotiates
in a way you cannot reproduce.

## Embedded JSON: the `json` locator

A `json` field evaluates a JSONPath over the JSON embedded in the page. Kaabist looks at
`<script id="__NEXT_DATA__">`, `window.__INITIAL_STATE__`, `application/ld+json` and
`application/json` script tags, in document order, and the first non-empty result wins:

```json
"extract": {
  "stock": { "json": "$.props.pageProps.product.stock", "type": "integer" },
  "list_price": { "json": "$.props.pageProps.product.price", "type": "number" }
}
```

A product page such as `https://shop.example.com/p/1` that embeds its product in `__NEXT_DATA__`
lets this extract read the stock and the list price there at the http tier, while other fields of the same extract read the
HTML.

## API responses: the `network` locator

A `network` field reads a JSON response the page received: `url` is a glob matched against the full
URL, `method` is optional, and `json` is the JSONPath into the body.

```json
"offers": { "network": { "url": "**/api/offers*", "json": "$.items[*].price" }, "type": "list" }
```

Give a path that returns several values `"type": "list"`. A list of objects works too: put the
`network` locator on a `many: true` field with `fields`, and each subfield's `json` path is then
evaluated against its own item (`$.price`), the way a CSS subfield is located inside its own card.

To be sure the response has arrived before you read it, make the navigation wait for it with a
`network` condition:

```json
{ "goto": "https://app.example.com/", "expect": { "network": { "url": "**/api/**", "status": 200 } } }
```

At the **browser tier** the responses are the ones the page fetched. At the **http tier** no script
runs, so Kaabist fetches the endpoints your `network` locators name itself, alongside the document.
It can only do that for a glob that names one endpoint:

- a leading `**/` is resolved against the page's own origin, so `**/api/products` fetches
  `/api/products` on the site you navigated to;
- a trailing `*` after the path (`**/api/offers*`) is dropped, which asks the endpoint for its
  defaults;
- a `*` anywhere inside the path (`**/api/**`, `**/products/*`) cannot be fetched, and that
  locator finds nothing at the http tier.

At most three endpoints are fetched per page, all at once, each within 5 seconds and up to 512 KB.
Only `GET` and `HEAD` endpoints are fetched: a `network` locator naming another `method` finds
nothing at the http tier, and the job reads it in a browser. Each endpoint is fetched once per job,
however many pages ask for it. An answer of 400 or above is not recorded, and one over 512 KB is
listed as truncated, with no body.

## Capturing a response: `capture_response`

`capture_response` waits for a matching response that arrives while the step runs and stores it,
or the part of it a JSONPath selects, in a variable:

```json
{ "capture_response": { "url": "**/api/more*", "method": "GET", "json": "$.items[*].label",
  "into": "vars.labels", "timeout_ms": 15000 } }
```

Later steps read it as `{{vars.labels}}`. Use it for something the page fetches in reaction to an
action (a "load more", a filter) or for a value you need in a URL or a condition rather than in the
result. The response body may be at most 2 MB.

## API discovery

After a successful browser-tier run, Kaabist compares the values the job extracted with the JSON
responses the page fetched. When one endpoint explains the extracted data (most top-level fields
and every list field found in it), it builds a `network` locator for each field, or a `json`
locator if the same JSON is embedded in the document. It then checks them by re-running the
extract with only the new locators against the same page, and discards them unless the result is
identical. Discovery runs at most once a day per program version.

What survives becomes a **proposal** of kind `api_discovery` on your program, waiting in the review
queue. Nothing changes until you approve it. Approving it:

- puts the discovered locators first in each field's `candidates`, keeping your previous locators
  after them as fallbacks;
- when **every** top-level field then reads JSON, sets `options.prefer_tier: "http"`, so the next
  job starts without a browser;
- with `promote: true`, publishes the result as a new version and moves the `latest` alias (never
  `production`, which you move yourself).

The program now reads the endpoint instead of the DOM, and runs at the http tier: the same data,
for the price of an http fetch.

## Worked example

Take a single-page app at `https://app.example.com` that is an empty shell: its script fetches the
products from a JSON API half a second after loading, then draws the cards. This program waits for that API call and reads the
product names and prices straight out of its body:

### TypeScript

```ts
const definition = {
  kaab: "1",
  name,
  options: { max_tier: "browser" }, // the API call is made by the page's JavaScript
  steps: [
    // Wait until the page's API call has answered…
    { goto: `${site}/`, expect: { network: { url: "**/api/**", status: 200 } } },
    {
      extract: {
        // …then read fields straight out of its JSON body.
        names: { network: { url: "**/api/**", json: "$..items[*].name" }, type: "list" },
        prices: { network: { url: "**/api/**", json: "$..items[*].price" }, type: "list" },
      },
    },
  ],
};
```

The job succeeds at the browser tier with `data.names` and `data.prices` read from the API.

> **Note:** A `network` field is the dependable way to read a response today. A `capture_response` step does
> not yet catch a response that arrives while it runs on a page like this one; until that is fixed,
> read the response with a `network` locator, as above.

To see discovery at work, publish a program that reads the same site with CSS locators only
(`#products .product-card` with `.title`, `.price` and `.sku` subfields) and `prefer_tier: "http"`:

1. The first job starts at the http tier, finds an empty shell, and escalates to the browser, where
   it succeeds.
2. A pending proposal of kind `api_discovery` appears for that program, with `network` locators on
   the site's `/api/products` endpoint.
3. Approve it with `promote: true`, and remove any `expect` on the navigation that counts cards in
   the DOM (see the pitfalls).
4. The next job reads the endpoint at the http tier and returns the same products.

## Pitfalls

- **DOM postconditions keep a job on the browser.** Approving a discovery rewrites locators, not
  `expect` conditions. A `goto` that still expects product cards in the HTML fails at the http
  tier, because the shell has none, and the job escalates again. Expect a `network` response
  instead, or nothing.
- **Wildcards inside the path.** At the http tier an endpoint is only fetched when the glob's path
  is literal. The worked example's `**/api/**` is fine in a browser and finds nothing without one;
  write `**/api/products` (or the full URL) for a program meant to run at the http tier.
- **APIs that change shape.** An API is a contract the site never promised you either. Keep the
  CSS locators as later `candidates` (discovery does this for you) so a changed payload falls back
  rather than failing.
- **`capture_response` only sees responses that arrive while it runs.** A response that came during
  an earlier step is gone; capture it in the step that triggers it, or read it with a `network`
  locator in the extract.
- **Analytics beacons.** A page sends its own data to tracking endpoints too. Make the glob specific
  enough to match the data API only.

## Reference

- Fields: [`json`](https://docs.kaab.ist/reference/dsl/fields/json), [`network`](https://docs.kaab.ist/reference/dsl/fields/network),
  [`many`](https://docs.kaab.ist/reference/dsl/fields/many)
- [`capture_response`](https://docs.kaab.ist/reference/dsl/actions/capture-response) and the
  [`network` condition](https://docs.kaab.ist/reference/dsl/conditions/network)
- [list proposals](https://docs.kaab.ist/reference/api/proposals/list-proposals), [approve a proposal](https://docs.kaab.ist/reference/api/proposals/approve-proposal),
  [proposal schema](https://docs.kaab.ist/reference/api/schemas/proposal)
- [Site profiles](https://docs.kaab.ist/reference/api/sites/get-site-profile) (`api_endpoints_count`)
- Concepts: [tiers and escalation](https://docs.kaab.ist/concepts/tiers-and-escalation), [the http tier](https://docs.kaab.ist/concepts/http-tier),
  [locators](https://docs.kaab.ist/concepts/locators)