20 Evaluation, Testing & Optimization Practice Questions & Answers

1. 1. A team rewrites a production prompt. The tech lead reads five outputs from the new version, says they look better, and asks to ship. What should the architect require before the change is accepted?

   • A.Ship the change and watch the production error-rate dashboard for a week
   • B.Have the lead score the same five outputs on a 1-5 scale so the improvement is quantified
   • C.Have three additional reviewers each read five outputs of their own choosing and take the majority opinion
   • D.Run both prompt versions against the same fixed evaluation set and compare the scores case by case✓ Answer

   A spot check draws a tiny, reviewer-selected, non-repeatable sample, so it cannot separate a real improvement from sampling noise or from the reviewer's expectation that the new version is better. Holding the input set fixed across versions is what makes the difference in scores attributable to the prompt change; quantifying or repeating an ad hoc sample does not fix the selection problem, and an error-rate dashboard measures availability, not output quality.

   Source: Anthropic Anthropic Claude certification (Architect – Professional), Domain 4 task statement 4.3 (conduct A/B testing and iterative improvements); Anthropic docs, 'Create strong empirical evaluations'Report a problem with this question

2. 2. An architect is building the first evaluation set for a customer-support assistant that has been live for three months. The team proposes brainstorming tricky inputs in a workshop. What is the better sourcing strategy?

   • A.Generate a large synthetic set of paraphrased inputs so the set can be produced quickly and grows automatically
   • B.Draw cases from logged production traffic and from real reported failures, covering the ordinary path as well as the edges, and add every newly reported failure as a permanent case✓ Answer
   • C.Include only cases the current system already handles correctly, so the suite stays green and can gate deploys
   • D.Include only the hardest adversarial inputs the team can devise, since ordinary requests already work

   An evaluation set is only informative if it mirrors the real task distribution, and imagined inputs systematically miss the failure modes users actually produce. Sourcing from production traffic and reported incidents both matches the real distribution and closes the feedback loop, so each production failure becomes a permanent regression case; a suite of only edge cases or only passing cases cannot detect a change in overall quality.

   Source: Anthropic docs, 'Create strong empirical evaluations' - eval design principles (task-specific, mirror real-world distribution incl. edge cases); Anthropic Claude certification (Architect – Professional), Domain 4 task statement 4.2Report a problem with this question

3. 3. An evaluation set contains 20 cases. The current prompt scores 18/20 and a candidate prompt scores 19/20. The team writes in the release notes that the change delivers a 5-point accuracy gain. What should the architect say?

   • A.Report the result as a range instead of a point estimate and ship the candidate either way
   • B.The claim is fine because both prompts were measured on exactly the same 20 cases
   • C.Re-run both prompts three times at a deterministic setting and report the average, which removes the uncertainty
   • D.The difference is one case and is indistinguishable from noise; the set must be large enough for the expected effect size before any gain is claimed✓ Answer

   With 20 cases, one item is worth five percentage points, so the observed 'gain' is within the range that resampling the dataset would produce by chance. Using the identical cases controls for dataset composition but not for the fact that the set is too small to resolve the effect, and repeating runs only reduces generation variance, not the sampling error of a 20-item set.

   Source: Anthropic docs, 'Create strong empirical evaluations' - prioritize volume of automatically graded cases; Anthropic Claude certification (Architect – Professional), Domain 4 task statement 4.2Report a problem with this question

4. 4. A team iterated a prompt for two weeks against a 200-case evaluation set, stopping when the score reached 96%. Leadership asks what accuracy to expect in production. What is the correct response?

   • A.Report 96%, since the 200 cases were drawn from real production traffic and are therefore representative
   • B.Measure on a held-out set that was never used during iteration; the 96% is optimistically biased because the prompt was tuned against those exact cases✓ Answer
   • C.Re-run the same 200 cases with a different random seed and report the lower of the two scores
   • D.Split the same 200 cases into two halves now and report the score on the second half as the unbiased estimate

   Repeatedly changing the prompt until a specific set of cases passes fits the prompt to that set, so its score stops being an estimate of performance on unseen inputs. Only cases withheld from the whole iteration loop give an unbiased estimate; splitting the set after the fact does not help, because both halves already influenced the tuning decisions.

   Source: Anthropic docs, 'Create strong empirical evaluations' - held-out evaluation data; Anthropic Claude certification (Architect – Professional), Domain 4 task statement 4.2Report a problem with this question

5. 5. A system routes each inbound message into exactly one of twelve fixed categories, and labeled ground truth exists for every evaluation case. The team currently grades this evaluation with a model-based judge. What should the architect change?

   • A.Keep the judge but instruct it to be stricter and to explain its reasoning
   • B.Keep the judge and add human review of a sample of its verdicts
   • C.Grade with embedding similarity between the predicted label and the true label
   • D.Replace the judge with deterministic exact-match grading against the labels✓ Answer

   The grader must match the nature of the task: for a closed label set with ground truth, correctness is a string identity check, so exact match is perfectly reliable, free of judge error, reproducible, and cheap enough to run on every change. A model-based judge introduces its own error rate and cost for a decision that requires no judgment, and similarity scoring can mark a wrong-but-related label as nearly correct.

   Source: Anthropic docs, 'Create strong empirical evaluations' - grading methods (code-based/exact match for deterministic tasks); Anthropic Claude certification (Architect – Professional), Domain 4 task statement 4.2Report a problem with this question

6. 6. A model produces structured records that a downstream billing service parses automatically. The evaluation grades outputs by semantic similarity to a reference record, and scores are high, yet the billing service still rejects several percent of records. What should the evaluation do instead?

   • A.Ask a model-based judge whether each output looks like a valid record
   • B.Add an automatic retry in the pipeline whenever the billing service rejects a record
   • C.Raise the semantic similarity threshold until the rejected records fall below it
   • D.Validate each output against the schema and assert required fields, types and value constraints, treating any violation as a failure✓ Answer

   For machine-consumed output, correctness is defined by the contract the consumer enforces, and a similarity score can rate a record highly while it is missing a required field or uses the wrong type. Structural and field-level validation is deterministic and detects exactly the defect that causes downstream rejection; retrying is a compensating control that hides the defect rather than measuring or removing it.

   Source: Anthropic docs, 'Create strong empirical evaluations' - match grading method to output type; Anthropic Claude certification (Architect – Professional), Domain 4 task statement 4.2Report a problem with this question

7. 7. A summarization system is graded by a model-based judge that runs on the same model, with the same domain system prompt, as the generator. Human audit finds the judge consistently approves summaries containing unsupported claims. What is the right correction?

   • A.Instruct the judge to be harsher and to reject anything it is unsure about
   • B.Run the same judge three times per case and take the majority verdict
   • C.Raise the judge's sampling temperature so its verdicts vary less predictably
   • D.Grade with a different model than the one being evaluated, give the judge the source document, and score each claim explicitly for support✓ Answer

   A judge built from the same model and context as the generator inherits the generator's blind spots and shows self-preference, so it cannot see the very errors the generator makes; repeating or hardening that judge repeats the same bias. Changing the grading model and grounding the judgement in the source document with a claim-level rubric gives the judge information and a perspective the generator did not have.

   Source: Anthropic docs, 'Create strong empirical evaluations' - LLM-based grading guidance (use a separate grading model; ground judgements in source); Anthropic Claude certification (Architect – Professional), Domain 4 task statement 4.2Report a problem with this question

8. 8. Two reviewers grade the same 60 open-ended responses and disagree on nearly half of them. What should the architect do first?

   • A.Average the two reviewers' scores, since averaging cancels individual bias
   • B.Adopt the more senior reviewer's scores as authoritative
   • C.Write an explicit rubric with anchored levels and worked examples, calibrate both reviewers on a shared subset, then re-measure agreement✓ Answer
   • D.Replace both reviewers with a model-based judge, which will at least be internally consistent

   High disagreement means the grading criteria, not the system, are undefined, and a score produced by an undefined criterion carries no information no matter how it is aggregated. Anchoring each level with examples and calibrating graders makes the measurement repeatable; a model-based judge inherits the same ambiguity because it would be given the same undefined criteria.

   Source: Anthropic docs, 'Create strong empirical evaluations' - rubric-based grading and consistency; Anthropic Claude certification (Architect – Professional), Domain 4 task statement 4.2Report a problem with this question

9. 9. A regulated workflow cannot afford human review of every generated output, but leadership will not accept unreviewed high-impact errors. What review design should the architect propose?

   • A.Review the first fifty outputs each morning, which spreads the effort evenly across the workload
   • B.Drop human review entirely and rely on a model-based judge running on all traffic
   • C.Review only the cases that users complain about, since those are the confirmed failures
   • D.Route low-confidence and high-impact cases to human reviewers, and additionally review a random sample of all traffic to keep an unbiased quality estimate✓ Answer

   Human attention is a scarce grading resource and should be spent where automation is least trustworthy and the consequence of error is greatest, which is what confidence- and impact-based routing does. The random sample is still required, because routed cases are a biased slice and only unbiased sampling can estimate the true quality rate; complaint-driven review measures what users noticed, not what went wrong.

   Source: Anthropic Claude certification (Architect – Professional), Domain 4 task statements 4.1 and 4.2 (mixed methodologies, human review where automation is insufficient); Anthropic docs on human-in-the-loop reviewReport a problem with this question

10. 10. A stakeholder states the requirement as: 'the assistant should sound helpful and rarely be wrong.' What is the architect's correct next step before any evaluation is run?

    • A.Defer the thresholds until after launch, when real user behaviour will show what matters
    • B.Run the evaluation first, then set the thresholds at whatever the system currently achieves
    • C.Convert it into measurable, agreed thresholds on named metrics and a named dataset, including quality, latency and cost, before the evaluation runs✓ Answer
    • D.Collapse everything into a single composite quality score so there is one number to approve

    Success criteria must be specific, measurable, achievable against a benchmark, and relevant, and they must be fixed before results are seen, otherwise the threshold is rationalized to whatever the system happened to produce. Most real applications need several criteria at once, because a single blended score lets a latency or cost regression be masked by a quality gain.

    Source: Anthropic docs, 'Define your success criteria' (SMART, multidimensional criteria); Anthropic Claude certification (Architect – Professional), Domain 4 task statement 4.1Report a problem with this question

11. 11. A new document-extraction pipeline scores 88% on its evaluation set. The team presents this as evidence to replace the existing rules-based extractor. What is missing from the argument?

    • A.The incumbent extractor's score on the identical evaluation set, without which 88% cannot be interpreted as better or worse✓ Answer
    • B.Confirmation that 88% was measured with the same random seed on both runs
    • C.A stated target of 90%, which is the conventional bar for production extraction
    • D.A published benchmark score for the same task to compare against

    An absolute score has no meaning without a reference point, because the difficulty of the evaluation set determines what a given number represents; only measuring the incumbent on the identical cases lets the difference be attributed to the new pipeline. External benchmarks describe different data, and an arbitrary round-number target is not evidence about this system.

    Source: Anthropic docs, 'Define your success criteria' - benchmark-based, achievable targets; Anthropic Claude certification (Architect – Professional), Domain 4 task statement 4.3Report a problem with this question

12. 12. In one release a team changed the prompt wording, the retrieval chunk size, and the grading rubric's strictness. The evaluation score rose 3 points and median latency doubled. What should the architect direct?

    • A.Re-run the experiment changing one variable at a time against a frozen evaluation set and a frozen rubric, so the gain and the regression can each be attributed✓ Answer
    • B.Add detailed request logging so the source of the latency increase becomes visible in production
    • C.Revert all three changes and abandon the release
    • D.Ship it, because the quality gain outweighs the latency cost on balance

    When several variables move together the delta is uninterpretable: the gain may come from one change while another causes the latency regression, and a rubric change makes the two scores non-comparable in the first place. Isolating one variable at a time against a frozen dataset and rubric restores attribution; adding logging is a detection step that does not tell you which change was responsible for the quality delta.

    Source: Anthropic Claude certification (Architect – Professional), Domain 4 task statement 4.3 (A/B testing and iterative improvements); Anthropic docs, 'Create strong empirical evaluations'Report a problem with this question

13. 13. A customer reports one wrong answer. An engineer adds a sentence to the system prompt that fixes exactly that case and verifies it. What must happen before the change ships?

    • A.Add the reported case to the suite and ship, since the fix was verified on the case it targets
    • B.Re-run the entire evaluation suite and confirm no other cases regressed, then add the reported case to the suite permanently✓ Answer
    • C.Ship behind an A/B test and let production metrics reveal any regression
    • D.Ship the change, because a single added sentence is too narrow to affect unrelated behaviour

    A prompt is global state: an instruction added to fix one case applies to every request, so it can change behaviour on cases nobody re-checked, which is precisely why a regression suite exists. Verifying only the motivating case generalizes from a sample of one, and discovering the regression through production A/B means real users absorb it first.

    Source: Anthropic Claude certification (Architect – Professional), Domain 4 task statement 4.3; Anthropic docs, 'Create strong empirical evaluations' - regression evaluation on every changeReport a problem with this question

14. 14. An agentic workflow reports 94% accuracy, but operations complains about incidents in which a run consumes budget for an hour without finishing, and about failed tool calls. Accuracy is computed over runs that completed. What should the architect add to the metric set?

    • A.A stricter p99 latency alert, which will fire on the long-running cases
    • B.A higher turn limit so long runs are given the chance to finish successfully
    • C.More frequent recomputation of the accuracy metric on completed runs
    • D.Non-termination rate, tool failure rate, and latency and cost per attempted task, since accuracy over completed runs excludes the failures being reported✓ Answer

    Computing accuracy only over completed runs is survivorship bias: the runs that hang or die on a tool error are removed from the denominator, so the metric stays high while the system fails users and burns budget. Measuring per attempted task, and tracking termination and tool-failure rates alongside quality, makes those failures visible; raising the turn limit increases spend on exactly the pathology being reported.

    Source: Anthropic Claude certification (Architect – Professional), Domain 4 task statements 4.1 and 4.6 (metrics beyond accuracy: latency, cost, operational signals)Report a problem with this question

15. 15. A claims-triage assistant is intended to replace a manual review process. The team wants credible evidence before any customer is affected. Which evaluation approach best fits?

    • A.Run the assistant in shadow on the same live cases the human process handles, discard its output, and compare the two decisions offline before any ramp✓ Answer
    • B.Route a randomly selected group of real customers to the assistant immediately and compare outcomes
    • C.Evaluate only on a synthetic set of constructed claims, because real claims contain personal data
    • D.Have the assistant re-decide last year's closed claims and report its agreement rate as the launch evidence

    A shadow run exposes the system to the true live distribution while its output is never acted on, so it produces a real head-to-head comparison against the incumbent process at zero customer risk. Routing real customers first accepts that risk before evidence exists, a synthetic-only set does not reflect the real distribution, and replaying old closed cases can be a useful supplement but does not capture current traffic.

    Source: Anthropic Claude certification (Architect – Professional), Domain 4 task statement 4.3 (offline eval, shadow evaluation, ramp with rollback path)Report a problem with this question

16. 16. A team proposes logging the full prompt and full response of every request, including customer records and any credentials passed in tool arguments, into the shared application log store. What should the architect require?

    • A.Log everything as proposed and restrict who can open the logging dashboard
    • B.Log only latency and error counts, since prompts and responses are always sensitive
    • C.Log the identifiers, metrics and step metadata needed for correlation, redact personal data and never log secrets, and capture full payloads only for a sampled, access-controlled trace store with a retention limit✓ Answer
    • D.Store a hash of each prompt and response so the content is preserved but unreadable

    Observability requires enough signal to reconstruct what happened, but persisting raw payloads copies regulated data and secrets into a system with weaker controls and unbounded retention, which is a new exposure rather than a monitoring benefit. Keeping identifiers and metrics at full coverage while sampling redacted payloads preserves diagnosability at proportionate risk; dashboard permissions do not protect data already written, and a hash destroys the diagnostic value entirely.

    Source: Anthropic Claude certification (Architect – Professional), Domain 4 task statement 4.6 and Domain 3 objective on observability at scale (sampling, redaction, retention)Report a problem with this question

17. 17. A multi-step agent intermittently returns a wrong final answer. Today the platform logs only the user input, the final output, and total latency. What instrumentation change most directly enables diagnosis?

    • A.Emit a span per step under one correlation ID, recording each tool call and its arguments, the retrieved chunks and their scores, the number of turns, and why the loop terminated✓ Answer
    • B.Increase the sampling rate of final outputs so more wrong answers are captured
    • C.Add assertions that the final output matches expected strings, and alert when it does not
    • D.Extend log retention so historical wrong answers remain available for longer

    A wrong final answer is a symptom whose cause lies in one of several intermediate steps, and no amount of additional final-output data localizes it. Step-level traces tied together by a correlation ID show which tool was called with what arguments and what context was retrieved, so the failing component can be identified; asserting exact output strings is the wrong primitive for a non-deterministic system and will alert constantly on correct answers.

    Source: Anthropic Claude certification (Architect – Professional), Domain 3 objective 'Analyze observability challenges and select monitoring strategies at scale'; Domain 4 task statement 4.6Report a problem with this question

18. 18. After a scheduled refresh of the source document corpus, a retrieval-augmented assistant begins returning fluent, confident answers that are factually wrong. Latency, the prompt, and the model version are unchanged. Where should the architect investigate first?

    • A.The retrieval and indexing step, verifying that the refresh re-indexed and re-embedded the corpus consistently and that the retrieved passages actually support the answers✓ Answer
    • B.The sampling temperature, lowering it until the answers become less confident
    • C.The model tier, moving to a more capable model to reduce factual errors
    • D.The system prompt, adding an instruction to say 'I don't know' when unsure

    Diagnosis follows what changed: the only altered component is the corpus, and the symptom of a confident answer built on the wrong context is exactly what a broken or partial re-index, or a corpus embedded inconsistently, produces. Temperature, prompt wording and model tier were all constant, so changing them treats a symptom of a defect located elsewhere, and an 'I don't know' instruction cannot help when the model is being handed context that looks authoritative but is wrong.

    Source: Anthropic Claude certification (Architect – Professional), Domain 4 task statement 4.4 (diagnose system issues); Anthropic guidance on retrieval failure vs. hallucination vs. model mismatchReport a problem with this question

19. 19. A provider dependency was upgraded six weeks ago. Uptime, error rate and latency dashboards stayed green throughout, but a customer audit now shows answer quality degraded from the week of the upgrade. What is the durable fix?

    • A.Make an automated regression evaluation on the golden set a required gate for every prompt, model, dependency or index change in the deployment pipeline✓ Answer
    • B.Add more operational alerts, including tighter latency and error-rate thresholds
    • C.Ask users to report answers that look wrong through an in-product feedback button
    • D.Increase log retention so future degradations can be traced further back

    Operational telemetry answers whether the system is up and responding; it cannot answer whether the output is correct, which is why the dashboards stayed green while quality drifted. Only an evaluation re-run on a fixed golden set detects a quality regression, and gating every change on it catches drift at the moment it is introduced instead of six weeks later; more alerts and longer retention measure the same availability signals, and user reports are lagging and biased.

    Source: Anthropic Claude certification (Architect – Professional), Domain 4 task statements 4.4 and 4.6 (evaluation detects correctness; telemetry detects availability); Anthropic docs on regression evaluationReport a problem with this question

20. 20. After five rounds of prompt iteration, an evaluation still sits at 62% and the failure cases cluster on one cause: the required facts are held in a system the assistant has no access to. Each iteration moved the score by about half a point. What should the architect conclude?

    • A.The evaluation is showing that the approach, not the wording, is the limit; the design must give the system access to the missing data before further tuning is worthwhile✓ Answer
    • B.Ship with a disclaimer telling users the assistant may lack current information, and log the failures
    • C.Continue prompt iteration, since every round has produced a measurable gain
    • D.Lower the success threshold to 62%, which is what the system demonstrably achieves today

    Evaluation results are diagnostic, not just a scoreboard: when failures cluster on a single root cause that no prompt can reach, the marginal return of further wording changes is bounded and the correct response is an architectural change that supplies the missing information. Relaxing the threshold to match observed performance rationalizes the criterion after the fact, and a disclaimer plus logging is a compensating control that neither prevents nor corrects the wrong answers.

    Source: Anthropic Claude certification (Architect – Professional), Domain 4 task statements 4.1 and 4.4 (success criteria set in advance; root-cause diagnosis over compensating controls)Report a problem with this question

Practice questions based on the official Claude Certified Architect – Professional (CCAR-P) exam guide and Anthropic's public documentation. This is an independent study tool, not affiliated with or endorsed by Anthropic, and does not grant certification. The real exam is 63 questions, 120 minutes, passing at a scaled 720/1000, delivered via Pearson VUE ($175). Official certification page →