JSON and network locators

Many pages are drawn from JSON: a server-rendered page embeds it in a script tag, a single-page app fetches it from an API after loading. Reading that JSON directly is more precise than reading the HTML made from it, survives redesigns, and can let a job run at the cheaper http tier without a browser.

#When to use it

• The data is in the page source as JSON (__NEXT_DATA__, window.__INITIAL_STATE__, application/ld+json): use a json locator. It works at every tier.
• The page fetches the data from an API while it loads: use a network locator, or a capture_response step when you need the response in a variable for later steps.
• You do not know where the data comes from: run the program at the browser tier with ordinary CSS locators, and let API discovery propose the JSON locators for you.

Stay with CSS locators when the values only exist in the rendered page (a price computed in the browser, text assembled from several responses), or when the API needs a token the page negotiates in a way you cannot reproduce.

#Embedded JSON: the json locator

A json field evaluates a JSONPath over the JSON embedded in the page. Kaabist looks at <script id="__NEXT_DATA__">, window.__INITIAL_STATE__, application/ld+json and application/json script tags, in document order, and the first non-empty result wins:

"extract": {
  "stock": { "json": "$.props.pageProps.product.stock", "type": "integer" },
  "list_price": { "json": "$.props.pageProps.product.price", "type": "number" }
}

A product page such as https://shop.example.com/p/1 that embeds its product in __NEXT_DATA__ lets this extract read the stock and the list price there at the http tier, while other fields of the same extract read the HTML.

#API responses: the network locator

A network field reads a JSON response the page received: url is a glob matched against the full URL, method is optional, and json is the JSONPath into the body.

"offers": { "network": { "url": "**/api/offers*", "json": "$.items[*].price" }, "type": "list" }

Give a path that returns several values "type": "list". A list of objects works too: put the network locator on a many: true field with fields, and each subfield's json path is then evaluated against its own item ($.price), the way a CSS subfield is located inside its own card.

To be sure the response has arrived before you read it, make the navigation wait for it with a network condition:

{ "goto": "https://app.example.com/", "expect": { "network": { "url": "**/api/**", "status": 200 } } }

At the browser tier the responses are the ones the page fetched. At the http tier no script runs, so Kaabist fetches the endpoints your network locators name itself, alongside the document. It can only do that for a glob that names one endpoint:

• a leading **/ is resolved against the page's own origin, so **/api/products fetches /api/products on the site you navigated to;
• a trailing * after the path (**/api/offers*) is dropped, which asks the endpoint for its defaults;
• a * anywhere inside the path (**/api/**, **/products/*) cannot be fetched, and that locator finds nothing at the http tier.

At most three endpoints are fetched per page, all at once, each within 5 seconds and up to 512 KB. Only GET and HEAD endpoints are fetched: a network locator naming another method finds nothing at the http tier, and the job reads it in a browser. Each endpoint is fetched once per job, however many pages ask for it. An answer of 400 or above is not recorded, and one over 512 KB is listed as truncated, with no body.

#Capturing a response: capture_response

capture_response waits for a matching response that arrives while the step runs and stores it, or the part of it a JSONPath selects, in a variable:

{ "capture_response": { "url": "**/api/more*", "method": "GET", "json": "$.items[*].label",
  "into": "vars.labels", "timeout_ms": 15000 } }

Later steps read it as {{vars.labels}}. Use it for something the page fetches in reaction to an action (a "load more", a filter) or for a value you need in a URL or a condition rather than in the result. The response body may be at most 2 MB.

#API discovery

After a successful browser-tier run, Kaabist compares the values the job extracted with the JSON responses the page fetched. When one endpoint explains the extracted data (most top-level fields and every list field found in it), it builds a network locator for each field, or a json locator if the same JSON is embedded in the document. It then checks them by re-running the extract with only the new locators against the same page, and discards them unless the result is identical. Discovery runs at most once a day per program version.

What survives becomes a proposal of kind api_discovery on your program, waiting in the review queue. Nothing changes until you approve it. Approving it:

• puts the discovered locators first in each field's candidates, keeping your previous locators after them as fallbacks;
• when every top-level field then reads JSON, sets options.prefer_tier: "http", so the next job starts without a browser;
• with promote: true, publishes the result as a new version and moves the latest alias (never production, which you move yourself).

The program now reads the endpoint instead of the DOM, and runs at the http tier: the same data, for the price of an http fetch.

#Worked example

Take a single-page app at https://app.example.com that is an empty shell: its script fetches the products from a JSON API half a second after loading, then draws the cards. This program waits for that API call and reads the product names and prices straight out of its body:

const definition = {
  kaab: "1",
  name,
  options: { max_tier: "browser" }, // the API call is made by the page's JavaScript
  steps: [
    // Wait until the page's API call has answered…
    { goto: `${site}/`, expect: { network: { url: "**/api/**", status: 200 } } },
    {
      extract: {
        // …then read fields straight out of its JSON body.
        names: { network: { url: "**/api/**", json: "$..items[*].name" }, type: "list" },
        prices: { network: { url: "**/api/**", json: "$..items[*].price" }, type: "list" },
      },
    },
  ],
};

The job succeeds at the browser tier with data.names and data.prices read from the API.

To see discovery at work, publish a program that reads the same site with CSS locators only (#products .product-card with .title, .price and .sku subfields) and prefer_tier: "http":

1. The first job starts at the http tier, finds an empty shell, and escalates to the browser, where it succeeds.
2. A pending proposal of kind api_discovery appears for that program, with network locators on the site's /api/products endpoint.
3. Approve it with promote: true, and remove any expect on the navigation that counts cards in the DOM (see the pitfalls).
4. The next job reads the endpoint at the http tier and returns the same products.

#Pitfalls

• DOM postconditions keep a job on the browser. Approving a discovery rewrites locators, not expect conditions. A goto that still expects product cards in the HTML fails at the http tier, because the shell has none, and the job escalates again. Expect a network response instead, or nothing.
• Wildcards inside the path. At the http tier an endpoint is only fetched when the glob's path is literal. The worked example's **/api/** is fine in a browser and finds nothing without one; write **/api/products (or the full URL) for a program meant to run at the http tier.
• APIs that change shape. An API is a contract the site never promised you either. Keep the CSS locators as later candidates (discovery does this for you) so a changed payload falls back rather than failing.
• capture_response only sees responses that arrive while it runs. A response that came during an earlier step is gone; capture it in the step that triggers it, or read it with a network locator in the extract.
• Analytics beacons. A page sends its own data to tracking endpoints too. Make the glob specific enough to match the data API only.

#Reference

• Fields: json, network, many
• capture_response and the network condition
• list proposals, approve a proposal, proposal schema
• Site profiles (api_endpoints_count)
• Concepts: tiers and escalation, the http tier, locators