Documents and PDFs

Some pages are not pages. A price list, a datasheet or a statement is often a PDF, reached by a link, a redirect or a download button. Kaabist treats every PDF as untrusted input: before it stores the file, reads its text, shows any of it to a model or gives you a link to it, the PDF passes a security gate. A file the gate refuses is never stored, and the step that met it fails with a code that says why.

#When to use it

Read this page when a job or an agent browser meets a PDF, when a download step fails with a pdf_* code, when you want a PDF's tables as data, or when you want to know what Kaabist does with documents before you point it at a site that publishes them.

#What the gate checks

Every PDF goes through the same four stages, whatever brought it in: a navigation that ended in a PDF, a download step, or an agent browser's download.

1. Is it a PDF? The file must start like one: %PDF- within its first kilobyte. A file that only claims to be a PDF is refused.
2. Malware scan. A malware scanner Kaabist runs in its own infrastructure checks the file; no file is sent to a scanning service. If the scanner cannot answer, the file is refused rather than passed unscanned.
3. Structure limits. The file's structure is measured before any parser reads it: at most 500 pages, 200 000 objects and 64 levels of nesting, and its compressed streams may expand to at most 50 MB each and 200 MB in total. Encrypted, password-protected files are refused.
4. Parsing in a sandbox. Reading the file's text happens in an isolated process with a memory and time limit and no network. A file that crashes it or runs out of either is refused, and nothing else is affected.

PDFs can carry scripts and actions. Kaabist never runs, follows or extracts them: scripts, open actions, rich media and forms are recorded as flags and left alone. A file that would launch a program, or that carries an executable attachment, is refused. Files over 64 MB, the limit for any download, are refused while they arrive.

#What you get back

• A navigation that ends in a PDF stores the file as the artifact source.pdf and records the step's snapshot as a PDF: it has no HTML, so an extract step over it returns null fields with the warning pdf_source_not_extractable instead of failing.
• A download step stores the PDF under its artifact name, as for any other file.
• An agent browser's download stores it the same way, and the tool answers with the block code when the gate refuses it.

#Reading a PDF's tables

A read_pdf step reads the tables of a PDF the job has stored: the page it is on when that page is a PDF, or a download's artifact by name. The file already passed the gate, so nothing is fetched or scanned again; it is parsed in the same sandbox.

{
  "kaab": "1",
  "inputs": { "url": "string" },
  "steps": [{ "goto": "{{inputs.url}}" }, { "read_pdf": {} }]
}

The reading lands in data.pdf:

{
  "artifact": "source.pdf",
  "sha256": "…",
  "tables": [
    { "id": "t1", "page": 1, "pages": [1, 2], "header": ["Code", "Description", "Price (TL)"], "rows": [["HW-100", "Copper wire 1.5 mm", "41,20"]] }
  ],
  "stats": { "pages": 2, "pages_read": 2, "tables": 1, "rows": 1, "no_text": false },
  "truncated": false
}

• Cells are verbatim strings. 41,20 stays 41,20: Kaabist does not parse numbers or dates out of a PDF, so you map the columns and their formats yourself.
• A table that continues over pages is one table, with every page it covers in pages. A header repeated at the top of each page is dropped; header is null when no first row reads as one. A row of column numbers (1, 2, 3, …) above or under the column titles is dropped, so the titles are the header.
• max_pages (1 to 500, the gate's page limit and the default) reads only the first pages of a long document. A document longer than that is read up to it with truncated: true and the warning pdf_pages_truncated, whose count and of say how many pages were read of how many.
• Headings inside a table are rows of it. Up to three lines with a single cell (a section heading such as NHXMH-J cables, or a description that wraps) between two parts of the same table are kept as rows: one non-empty cell, in the column where the text starts, every other cell "". A single-cell line above a table's first row or below its last is not a row, and a heading followed by a new header row starts a new table; include_text shows those lines.
• min_table_rows (1 to 20, default 3) is the fewest lines with two or more cells a table needs. Set 2 to keep a header with one row, or 1 for any line that splits into cells; single-cell rows are not counted.
• include_text: true adds each page's lines under text, to see the headings above a table.
• into: "vars" with a field keeps the reading for later steps instead of returning it.

A scanned PDF succeeds with no tables and the warning pdf_no_text. A step with nothing to read (the page is not a PDF, or the job stored no artifact by that name) fails pdf_source_missing. A large reading, such as a 300-page price book of 20 000 rows, is returned like any large result.

Each read_pdf is billed by the pages it read: one credit for every started 50 pages (config/pricing.yaml, pdf_read), counted as compute and shown as PDF reading in the job's cost lines. A replay of a stored run reads the stored PDF again at no charge.

A PDF that passes the gate but has no text at all, because its pages are blank or scanned images, is reported with the warning pdf_no_text. Kaabist does not run OCR: to get data out of a PDF, use a read_pdf step, and expect no tables from a scanned file.

#When a PDF is refused

The step fails with the code and nothing is written: there is no artifact to fetch and no link to share. The full list of codes is in the codes reference.