Somewhere in most businesses, a person opens a supplier PDF, reads the account number, the billing period and the total, and types them into a spreadsheet. Then they do it again for the next forty. It is slow, it is error-prone, and when the auditor asks where a figure came from, the honest answer is "someone typed it". The obvious fix is to hand the PDFs to an AI model, and for a lot of firms that is exactly where the conversation ends: supplier contracts, patient letters and quality data are not going to a cloud API.
So I built Echdyn, from the Welsh echdynnu, "to extract". It is a self-hosted tool that turns PDFs and scanned paperwork into validated, structured data using local models through Ollama. You describe a document type as a schema, Echdyn reads each upload, checks what it read, sends anything uncertain to a person, and publishes approved records through a REST API, signed webhooks and CSV or Excel exports. Nothing leaves your own hardware.
Getting a small local model to return JSON is the easy part. The interesting question, and the one this post is about, is when you can trust what it returned.
Why energy bills
The demo document type is the UK business energy bill, and the choice was deliberate. Every business gets them, every supplier lays them out differently, and, most usefully, the figures check themselves. Consumption times the unit rate, plus the standing charge for each day of the period, gives the subtotal. Subtotal plus VAT gives the total. An electricity meter number (the MPAN) even carries its own check digit.
That means a wrong answer is usually detectable without knowing the right one. Invoices, delivery notes and certificates of conformity have the same property to varying degrees, which is why I think arithmetic is the most underused tool in LLM extraction.
Make the model copy, not think
Ollama accepts a JSON Schema in its format field and compiles it into a grammar, so the model physically cannot return anything outside that shape. Echdyn defines each document type in Zod, converts it with Zod 4's built-in z.toJSONSchema, stores the versioned schema in Postgres, and passes it on every call:
const response = await client.chat({
model,
messages,
// Only the text path keeps grammar-forced ISO dates; anything read from an image copies them
format: toPrintedSchema(input.jsonSchema, { dates: source.kind !== "text" }),
stream: false,
options: { temperature },
});
const rawOutput = response.message.content;
const object = parseObject(rawOutput);
// raw_output keeps the values as printed; parsed holds them in the schema's units
const parsed = object ? normalisePrinted(input.jsonSchema, object, input.units) : null;
The toPrintedSchema call is the lesson from the first evaluation run, and it is worth explaining because it went against my instinct.
What went wrong first
My first schema asked for numbers as JSON numbers and dates as YYYY-MM-DD, with a prompt telling the model not to calculate anything. On 30 synthetic bills, llama3.2 scored 84.2% of fields correct. The misses were instructive:
- one supplier prints the unit rate in pounds per kWh, so the model wrote
0.183where the schema wanted18.3pence, because converting would have been calculating - number-constrained decoding mangled thousands separators:
£1,526.95came back as1.52695, because the grammar would not allow the comma - the vision model, forced into ISO dates while reading "10 February 2026", wrote the digits in page order and produced
1002-02-26
The fix was to stop asking a 3B model to do formatting. The schema sent to Ollama now asks for numeric fields as strings, constrained to look like a printed value (an optional £, one number, a unit or two), and dates as printed on scans. The model copies £1,526.95 and 18.3p exactly as it sees them, and deterministic code parses them back into the schema's units. A bare rate under 1 is pounds with the symbol dropped, since no UK rate is below a penny. Text accuracy went from 84.2% to 96.7%, and the first vision run went from 73.3% to 93.3%.
Small models are good at reading and bad at converting. Let them read.
Arithmetic is the real validator
Zod checks the shape: required fields, types, ranges. Then each document type carries business rules, and each rule names the fields it involves so the review screen can highlight them:
{
name: "vat-amount",
fields: ["subtotal_gbp", "vat_rate_percent", "vat_gbp"],
check: (bill) => {
const expected = (bill.subtotal_gbp * bill.vat_rate_percent) / 100;
return approxEqual(expected, bill.vat_gbp, ROUNDING_TOLERANCE_GBP)
? null
: `${bill.vat_rate_percent}% VAT on ${gbp(bill.subtotal_gbp)} is ${gbp(expected)}, but VAT is ${gbp(bill.vat_gbp)}`;
},
},
{
name: "total-adds-up",
fields: ["subtotal_gbp", "vat_gbp", "total_gbp"],
check: (bill) => {
const expected = bill.subtotal_gbp + bill.vat_gbp;
return approxEqual(expected, bill.total_gbp, ROUNDING_TOLERANCE_GBP)
? null
: `Subtotal + VAT = ${gbp(expected)}, but total is ${gbp(bill.total_gbp)}`;
},
},The messages are written for the reviewer, not the developer. "20% VAT on £322.40 is £64.48, but VAT is £386.88" tells a person exactly where to look on the PDF beside it. A model output that fails to parse at all is not thrown as an error either: the raw output is stored and the document goes to review. Only infrastructure failures, such as Ollama being down, throw and trigger a queue retry, so a quality problem never burns retries.
Reading twice
Rules catch values that do not fit together. They cannot catch a supplier name read as the customer's name, so Echdyn also reads every document twice: once at temperature 0, once at 0.4. At temperature 0 a model is effectively deterministic, so two identical runs would always agree. The second run samples a little, and fields the model is unsure about tend to wobble. Anything that failed a rule in either run, or changed between runs, is flagged:
export function scoreExtractions(runs: readonly Extraction[]): FieldError[] {
const flags = runs.flatMap((run) =>
run.validationErrors.map((error) => ({
...error,
message: `Run ${run.runNumber}: ${error.message}`,
})),
);
const parsed = runs
.map((run) => run.parsed)
.filter((p): p is Record<string, unknown> => p !== null);
// Compare only when every run produced an object; an unparsed run is already flagged
if (parsed.length === runs.length && parsed.length > 1) {
for (const field of mismatchedFields(parsed)) {
const values = parsed.map((p) => JSON.stringify(p[field] ?? null)).join(" vs ");
flags.push({ field, rule: RUN_MISMATCH_RULE, message: `Runs disagree: ${values}` });
}
}
return flags;
}No flags means the document can be approved automatically. Any flag sends it to a review screen with the PDF on one side, the extracted fields on the other and the reason for each flag at the top.
The two checks are not redundant. In one of the first live runs, llama3.2 read the VAT line as the total, and it did so identically in both runs. Consistency alone would have approved it. The VAT and total rules caught it. Equally, the double read catches misreadings that happen to be arithmetically harmless. Each catches failures the other misses, which is why both exist.
Measured, not claimed
Accuracy numbers for LLM tools are usually a feeling. I wanted a table, so Echdyn has an eval harness that runs labelled PDFs through the real pipeline (same stages, temperatures and rules as production, in its own database and bucket) and reports per field, per layout and per set:
$ make eval MODELS=llama3.2
| Set | Read by | Field accuracy | Fully correct | Flagged | Median latency |
| --------- | ------------ | -------------- | ------------- | ------- | -------------- |
| synthetic | llama3.2 | 96.7% | 66.7% | 33.3% | 32.5 s |
| scanned | qwen2.5vl:3b | 93.3% | 40.0% | 80.0% | 334.8 s |
Auto-approved with an error: 0 of 21The last line is the number I care about most. It counts documents that passed every check and still held a wrong value: the mistakes nobody would ever see. Field accuracy tells you how often the model is right. That line tells you whether you can trust the documents it did not ask you about. With 3B models on a laptop, every mistake was caught and sent to review. The cost is that 40% of documents needed a person, which is a cost you can see and plan for.
Those runs were on a laptop with no GPU, 8 GB of RAM and 4 cores given to WSL, so only 3B models fitted and a scan took over five minutes. I will rerun it on my home server with a GPU and larger models (qwen3:8b for text, qwen2.5vl:7b for scans) for comparison. The harness works on anyone's own labelled documents, so you can measure a model against your paperwork before trusting it.
The parts that make it usable
The model is a small fraction of the code. The rest is what turns a demo into something you could leave running. It is TypeScript throughout in a pnpm workspace: a Next.js 16 app for the UI and API, a Node worker running BullMQ jobs on Redis, Postgres through Drizzle, and S3-compatible storage. Coming from Laravel, BullMQ felt familiar, much like queued jobs with retries and backoff. The pipeline itself lives in a shared core package rather than the worker, which is what lets the eval harness run exactly the production stages without Redis.
- Idempotent uploads. Files are hashed with SHA-256 and stored under that hash, and a unique index on the document type and hash is the real duplicate guard, so a retried upload returns the original document
- Per-page reading. pdfjs classifies each page as text, scan or blank. Text goes to a text model, scans are rendered to PNG for a vision model, and mixed documents send both in one request
- Honest progress. Local models are slow, so the list shows each document's stage, its queue position and an estimate from recent runs instead of a spinner
- A full audit trail. Every approved value links to its source PDF, the model run that read it (model, prompt version, raw output) and any reviewer correction
- Signed webhooks. Approval writes the delivery to an outbox in the same transaction, then a separate queue POSTs it with a Stripe-style HMAC signature and retries for about an hour
- Document types as configuration. Variable fields live in JSONB with GIN indexes, so one set of tables serves every type, and the API can filter on any field, such as every bill over 30p per kWh
One small infrastructure note: MinIO stopped publishing Docker images during the build, so local storage is RustFS, which speaks the same S3 API. The app only ever talks S3, so production can use AWS S3 with a configuration change.
Summary
A small local model will make mistakes. The design question is whether those mistakes reach your data. Echdyn's answer has four parts: constrain the output shape with JSON Schema, let the model copy values as printed and parse them in code, check the result with rules the document itself obeys, and read twice to catch uncertainty. Then measure the whole thing, and watch the silent error count rather than the headline accuracy.
Echdyn runs with one docker compose up, Ollama included, and the showcase site walks through the screens and results. The code is on GitHub. If your team is retyping PDFs and would rather the data stayed on your own hardware, get in touch via the contact section.