Validate claims, not confidence

The problem

A data extraction pipeline processes financial figures from legal contracts and emits perfectly structured JSON: keys match, types align, and schema validation passes with zero errors. The team deploys the pipeline to production because "the output validates." Three weeks later, they discover that the total_value field was systematically wrong across contracts containing indexation adjustment clauses — the JSON was always syntactically valid, but the extracted numerical value inside it was incorrect from day one.

The pipeline validated output structure, but never validated output content. These are two completely distinct failure modes, and conflating them is the most expensive mistake in this domain.

Schema correctness is not factual correctness

There are two completely independent classes of failure:

• Structural failure: The output violates the expected schema — a missing key, an invalid datatype, or unparseable JSON syntax. This is detectable deterministically using an automated schema validator without domain knowledge.
• Content failure: The output adheres to the schema flawlessly, but the underlying data is wrong, incomplete, or hallucinated. Catching this requires domain verification — no generic schema validator can flag a plausible-looking but factually incorrect number.

A pipeline that validates only schemas is blind to content failures, which are frequently the costliest because they pass downstream under the false pretense that the output "looks correct."

When an output requires human review

Not every model output warrants human-in-the-loop review — enforcing manual review universally destroys the unit economics of automation. The right architectural question is not "could this output be wrong?" (almost any output can), but rather: if an error occurs, is the cost of failure high enough and the supporting evidence weak enough to justify manual review?

Two primary vectors govern this decision:

1. Cost of error: An internal draft reviewed downstream by a human editor carries low error cost. A factual assertion published directly to customers as verified truth, or an irreversible action triggered autonomously, carries high error cost.
2. Strength of evidence: An extracted value that appears verbatim in the source document carries strong evidence. An inference synthesized across disparate clauses carries weaker evidence, no matter how convincing the model sounds.

High error cost paired with weak supporting evidence is the textbook trigger for human-in-the-loop verification before execution. Low error cost with strong evidence can proceed autonomously. The prevalent anti-pattern is deciding based on "how confident the text sounds" — models generate hallucinations with the exact same fluent confidence as verified facts. Linguistic confidence is not empirical evidence of correctness.

Logging errors so root causes survive

When an output is flagged as erroneous — whether by human review or automated assertions — what you capture in the log is as crucial as catching the error itself. A telemetry log that merely records "this output was incorrect" is useless months later: context is lost, and the same architectural blind spot recurs.

An actionable error record captures at least three elements:

1. The specific constraint violated (not just "it was wrong", but "ignored annual inflation adjustment clauses").
2. What made the incorrect distractor plausible to the model or classifier.
3. The deterministic rule, test case, or guardrail that prevents recurrence.

This third element is what transforms error logging into continuous architectural improvement, rather than an unmaintainable pile of incident tickets.

Where intuition fails

• "If the JSON passes schema validation, the extraction is correct." Confuses structural integrity with semantic accuracy — passing schema validation provides zero guarantee of truthfulness.
• "A confident-sounding response is likely accurate." Fluency is an artifact of token prediction, not factuality; evidentiary strength originates from source grounding, not grammatical polish.
• "Requiring human review for everything is always the safest policy." Universal manual review destroys operational scalability; the correct posture calibrates review proportionally to error cost and evidence strength.

Put it into practice

In this lesson's lab, you will implement a two-stage validator — one validating schema structure and another evaluating content ground-truth signals — paired with an escalation function that routes outputs to automated execution or human review based on error cost and evidence strength.