Last in the 6-part CCAR-F scenario series — full source for every scenario: labs.
The scenario, as written in the exam guide
A system extracts information from unstructured documents, validates the output against JSON schemas, and keeps precision high while handling edge cases and integrating with downstream systems.
| Domain | Weight | Where it shows up in this scenario |
|---|---|---|
| Prompt Engineering & Structured Output | 20% | A tool_use JSON Schema, plus a second validation layer that catches what the schema can't |
| Context Management & Reliability | 15% | What happens downstream when a field can't be confidently extracted |
The obvious reading of this scenario is incomplete
"Use tool_use to get structured JSON out of Claude" is table stakes — necessary, and also not what separates a pass from a fail here. The exam guide's own phrase, keeps precision high while handling edge cases, is the actual test: what does the system do when a document is ambiguous, not when it's clean?
A JSON Schema can enforce shape. It cannot enforce honesty.
labs's EXTRACTION_TOOL_INPUT_SCHEMA is a plain JSON Schema handed to Claude via tool_choice: vendor is string | null, fieldsNeedingReview is an array of field names. That guarantees the response has the right shape. It does not — and structurally cannot — guarantee the response is internally consistent. Nothing in JSON Schema can express "if vendor is null, "vendor" must also appear in fieldsNeedingReview." That's a relationship between two fields, and JSON Schema validates fields individually.
ExtractionSchema (a zod schema, checked after the tool_use response comes back) adds exactly that check via superRefine: any nullable field that's null but not listed in fieldsNeedingReview fails validation. This is the whole scenario in one sentence: a schema alone doesn't stop a model from returning a confident-looking null it never flagged as uncertain — a second layer, checking a relationship the first layer structurally can't express, does.
Two failure modes, one fail-safe destination
route.ts only allows auto_process when validation passed and fieldsNeedingReview is empty. Both an honest "some fields are ambiguous" result and an invariant-violating "this claims confidence it didn't earn" result land in human_review — and critically, the invariant-violation case returns result: null to the caller rather than trying to salvage part of a self-contradictory extraction. Guessing which half of a broken result to trust is worse than admitting none of it can be trusted. Same "degrade to the safe state, don't propagate a broken value" instinct as Scenario 5's malformed-finding handling — here the cost of guessing wrong is a downstream system silently acting on bad data, not a missed PR comment.
Three fixtures, three different ways to be ambiguous
- Clean — unambiguous vendor, date, currency, total. Should auto-process.
- Ambiguous — a date that could be two different days (
03/04/2026), an illegible vendor, no currency symbol anywhere. - Multi-currency — line items in inconsistent currencies and an explicit "see final settlement invoice" instead of a total, testing whether the model flags
totalAmountrather than summing mismatched units into a number that looks fine but means nothing.
Run it yourself
git clone https://github.com/aicertstudy/labs
cd labs/ccar-f/scenarios/scenario-6-structured-data-extraction
npm install
npm test # 8 passing tests — the invariant and fail-safe routing, no API key needed
cp .env.example .env # add your ANTHROPIC_API_KEY
npm run extract -- fixtures/invoice-ambiguous.txt
npm test alone fully exercises the part worth understanding first — the invariant and the routing fail-safe — with zero API calls.
Test yourself
1. Why can't the tool_use JSON Schema alone enforce that a null vendor is always listed in fieldsNeedingReview?
Answer: JSON Schema validates each field's shape independently; it has no construct for a conditional relationship between two different fields' values.
"If field A is X, field B must contain Y" is a cross-field constraint, not a type constraint on any single field. Plain JSON Schema (and by extension a tool_use input_schema) can describe what each field individually looks like, not a rule spanning two of them.
2. What does route() return when the raw tool_use input violates the null/fieldsNeedingReview invariant — a best-effort partial result, or nothing?
Answer: nothing — result: null, routed to human_review, with no attempt to salvage or partially trust the self-contradictory data.
An internally inconsistent extraction doesn't tell you which of its claims to believe. Returning a partial or "best guess" result would require deciding which half of a broken claim is the trustworthy half — a decision the system has no principled basis for making.
3. In invoice-ambiguous.txt, why does the date 03/04/2026 deserve a fieldsNeedingReview flag even though a date IS extracted?
Answer: because the format is genuinely ambiguous (day-first vs. month-first), so any single value chosen is a guess dressed up as an extraction, not a confident read.
The invariant only catches null-but-unflagged cases mechanically; catching "extracted a value, but the value itself is a coin flip" depends on the prompt asking for exactly this (see extract.ts's system prompt: flag a field even when non-null if you're not confident it's correct) — the schema can enforce the null case, but a genuinely ambiguous non-null case relies on the model following that instruction.
4. Why does invoice-multi-currency.txt test something meaningfully different from invoice-ambiguous.txt?
Answer: it tests whether the model avoids silently combining mismatched units into a number that looks valid but is meaningless, rather than testing whether it can spot a single missing/illegible field.
Ambiguous-but-present data (an illegible vendor) and inconsistent-but-present data (mixed currencies summed as if they were one) are different failure shapes. The first is about recognizing absence; the second is about recognizing that a computed value can be schema-valid (a well-formed number) and still be substantively wrong.
5. If you deleted the superRefine check from ExtractionSchema, which specific test in route.test.ts would start failing, and why?
Answer: the "fails safe to human review... when the raw extraction violates the schema invariant" test — because without superRefine, a null vendor with an empty fieldsNeedingReview would pass validation and get routed to auto_process instead of human_review.
That test constructs exactly the case the invariant exists to catch. Remove the check that catches it, and the schema silently accepts what used to be a validation failure — which is precisely the failure mode the whole scenario is about.
Free CCA-F practice exam, covering all 5 domains: aicert.study/certifications/ccaf.