# Human review workflows and confidence calibration

*Claude Certified Architect - Foundations Study Guide · Context Management & Reliability · Last updated October 22, 2026 · https://www.certifyforce.com/study-guide/claude-certified-architect-foundations/context-management-and-reliability/human-review-workflows-and-confidence-calibration*

When it is safe to cut human review of extractions: segment-level accuracy instead of one aggregate number, field-level confidence calibrated on labeled data, routing scarce reviewers to low-confidence and contradictory records, stratified sampling of auto-approved output, and calibrating LLM graders against experts.

- Certification: Claude Certified Architect - Foundations
- Exam section: Context Management & Reliability
- Study guide: https://www.certifyforce.com/study-guide/claude-certified-architect-foundations

**Official exam objective:**

> Design human review workflows and confidence calibration

## What the exam expects

- Reject a proposal to remove human review on the strength of one aggregate accuracy figure, and ask for accuracy by document type and by field first.
- Have the model output field-level confidence scores, then set per-field review thresholds calibrated on a labeled validation set.
- Route low-confidence fields and ambiguous or contradictory source documents to human review first when reviewer capacity is limited.
- Keep a stratified random sample of high-confidence, auto-approved output under human review to measure error rates and catch novel error patterns.
- Calibrate an LLM grader against expert labels, give it an "Unknown" way out, and keep periodic human checks.

Key terms: **segment** (a slice of traffic such as one document type or one field); **labeled validation set** (documents with human-verified correct values, held out from tuning); **calibration** (checking that a confidence score matches observed accuracy); **stratified random sample** (a random sample drawn separately from each segment); **LLM-as-judge** (a model that grades outputs against a rubric).

## Key facts

| Fact | Detail | Source |
|---|---|---|
| Aggregate risk | 97% overall can mask poor accuracy on one document type or field | [Exam guide](https://everpath-course-content.s3-accelerate.amazonaws.com/instructor%2F6nizmqk8tpzpfjvt6qmmav7rh%2Fpublic%2F1783542750%2FClaude+Certified+Architect+%E2%80%93+Foundations+Exam+Guide.pdf) |
| Before automating | Validate accuracy by document type and field segment | [Exam guide](https://everpath-course-content.s3-accelerate.amazonaws.com/instructor%2F6nizmqk8tpzpfjvt6qmmav7rh%2Fpublic%2F1783542750%2FClaude+Certified+Architect+%E2%80%93+Foundations+Exam+Guide.pdf) |
| Threshold basis | Field-level confidence calibrated on a labeled validation set | [Exam guide](https://everpath-course-content.s3-accelerate.amazonaws.com/instructor%2F6nizmqk8tpzpfjvt6qmmav7rh%2Fpublic%2F1783542750%2FClaude+Certified+Architect+%E2%80%93+Foundations+Exam+Guide.pdf) |
| Review first | Low-confidence fields and ambiguous or contradictory sources | [Exam guide](https://everpath-course-content.s3-accelerate.amazonaws.com/instructor%2F6nizmqk8tpzpfjvt6qmmav7rh%2Fpublic%2F1783542750%2FClaude+Certified+Architect+%E2%80%93+Foundations+Exam+Guide.pdf) |
| Ongoing monitoring | Stratified random sample of high-confidence extractions | [Exam guide](https://everpath-course-content.s3-accelerate.amazonaws.com/instructor%2F6nizmqk8tpzpfjvt6qmmav7rh%2Fpublic%2F1783542750%2FClaude+Certified+Architect+%E2%80%93+Foundations+Exam+Guide.pdf) |
| Eval coverage | Mirror the real task distribution and include edge cases | [Develop tests](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests) |
| LLM grading order | Test reliability first, then scale | [Develop tests](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests) |
| LLM grader way out | Instruct it to return "Unknown" when information is insufficient | [Demystifying evals](https://www.anthropic.com/research/demystifying-evals-for-ai-agents) |
| After a prompt or model change | Rerun evals on each agent change and model upgrade; recalibrate | [Demystifying evals](https://www.anthropic.com/research/demystifying-evals-for-ai-agents) |
| Robust grader | Human review only occasionally, not never | [Demystifying evals](https://www.anthropic.com/research/demystifying-evals-for-ai-agents) |
| User feedback as a signal | Sparse, self-selected, skews toward severe issues | [Demystifying evals](https://www.anthropic.com/research/demystifying-evals-for-ai-agents) |

## Check yourself

- Why can a pipeline with 97% overall accuracy still be unsafe to automate?
- When reviewers can check only a fraction of daily volume, which records go first?
- Why sample high-confidence output that has already been auto-approved?
- What should an LLM grader be allowed to answer when it lacks information?

## Sources

- [Claude Certified Architect – Foundations Exam Guide v1.0](https://everpath-course-content.s3-accelerate.amazonaws.com/instructor%2F6nizmqk8tpzpfjvt6qmmav7rh%2Fpublic%2F1783542750%2FClaude+Certified+Architect+%E2%80%93+Foundations+Exam+Guide.pdf)
- [Define success criteria and build evaluations](https://platform.claude.com/docs/en/test-and-evaluate/develop-tests)
- [Demystifying evals for AI agents](https://www.anthropic.com/research/demystifying-evals-for-ai-agents)
- [Reduce hallucinations](https://platform.claude.com/docs/en/test-and-evaluate/strengthen-guardrails/reduce-hallucinations)
- [Best practices for Claude Code](https://code.claude.com/docs/en/best-practices)
- [How we built our multi-agent research system](https://www.anthropic.com/research/multi-agent-research-system)

## About the full unit

The "Core concepts", "Exam traps" and "Worked scenario" sections of this unit, its 7 linked practice questions, and unlimited timed mock exams are included with the Claude Certified Architect - Foundations pack (USD $19): https://www.certifyforce.com/practice-exams