How to test answer relevance and grounding

An answer can sound fluent while misstating a candidate's experience. This method evaluates relevance, factual grounding and usefulness as separate qualities. It contains no product scores or outcome claims.

The test material#

Create a fictional role description and a short fictional candidate profile with a fixed set of facts: project responsibilities, a decision, a trade-off and an observed result. Include one deliberate gap, such as an absent performance figure. Publish the full inputs, permitted source boundaries and version of each tool or model before running the test. Use no real applicant data or confidential assessment questions.

The same tasks for each run#

Ask every system to answer the same first question and two follow-ups. Keep the prompt wording, context, language, time limit and allowed interactions equal. Run a no-context baseline as well as the context-supplied condition. Save each raw answer and the actual setup steps so another person can reproduce the comparison.

Score what the answer says#

Have two independent reviewers, unaware of which system produced an answer, mark each material claim as supported, contradicted or unsupported by the supplied facts. Separately score whether the answer addresses the question, explains the candidate's decision and handles the missing figure honestly. Resolve disagreement by quoting the input fact. Report all answers, failures, reviewer rules, model versions and sample size alongside any aggregate; do not turn a small exercise into a hiring-success rate.

You can apply this checklist to your own interview practice and technical story bank. For a personal answer, trace every claim back to a project note before practicing it aloud.

An original test fixture#

Role: support operations coordinator. Fixed candidate facts: mapped ticket handoffs, introduced an owner checklist, and observed fewer unresolved handoffs in a team review. Missing fact: no numeric before-and-after measurement was supplied.

Ask: “Describe a process you improved.” Follow with: “How much did it improve?” and “What other explanation might account for the change?” A percentage is unsupported by this fixture; a response describing the observed review without a percentage can still be relevant. This is proposed test material, not a reported run. Try the preparation preview with the fixture and mark claims using the evidence checklist.