Human review workflows and confidence calibration

What the exam expects

• Reject a proposal to remove human review on the strength of one aggregate accuracy figure, and ask for accuracy by document type and by field first.
• Have the model output field-level confidence scores, then set per-field review thresholds calibrated on a labeled validation set.
• Route low-confidence fields and ambiguous or contradictory source documents to human review first when reviewer capacity is limited.
• Keep a stratified random sample of high-confidence, auto-approved output under human review to measure error rates and catch novel error patterns.
• Calibrate an LLM grader against expert labels, give it an "Unknown" way out, and keep periodic human checks.

Key terms: segment (a slice of traffic such as one document type or one field); labeled validation set (documents with human-verified correct values, held out from tuning); calibration (checking that a confidence score matches observed accuracy); stratified random sample (a random sample drawn separately from each segment); LLM-as-judge (a model that grades outputs against a rubric).

Key facts

Check yourself