The automated scores were too close to decide
Both agents followed the required response format. Both retrieved from the same product documentation. Automated checks produced similar averages.
The product team still needed to know:
- Which answer understood the user's actual problem?
- Which next step could a customer follow?
- Which response admitted uncertainty?
- Which one promised an action the policy did not allow?
- Which answer knew when to escalate?
A model judge can contribute signal. Using it as the only judge risks rewarding familiar style or surface completeness.
The evaluation is designed before outputs are scored
The company freezes 30 prompts drawn from anonymized support patterns. Each agent answers under the same context, tool access, temperature, and policy.
Before Taskin receives the package:
- model names are removed;
- response order is randomized per prompt;
- prompt context is identical;
- personal and customer data is removed;
- expected factual references are supplied;
- the rubric and critical-error definitions are fixed.
The evaluator does not know which system produced A or B.
The human rubric
For each answer, the participant scores:
- factual correctness;
- relevance;
- actionability;
- clarity;
- appropriate uncertainty;
- policy compliance;
- escalation quality;
- unsupported promises.
For each pair, the participant selects A, B, or tie and writes a short evidence-based reason. A critical error must quote the relevant output and name the violated fact or policy.
The task does not ask which answer sounds more polished.
Structured result
prompt_id: support_017
response_a:
correctness: <score>
relevance: <score>
actionability: <score>
uncertainty: <score>
response_b:
correctness: <score>
relevance: <score>
actionability: <score>
uncertainty: <score>
preference: A, B, or tie
reason: <evidence-based explanation>
critical_error:
response: <A, B, or none>
passage: <exact text>
rule: <fact or policy>
evaluator_confidence: <defined scale>Acceptance test
The evaluation is complete when:
- 1.every prompt receives both score sets;
- 2.every preference includes a reason;
- 3.critical errors quote the output;
- 4.model identity remains hidden;
- 5.the evaluator uses only supplied facts and policy;
- 6.incomplete or ambiguous cases are marked;
- 7.no row is silently skipped.
How the result enters the model decision
The experiment agent aggregates preference rate, mean score by dimension, critical errors, and uncertainty. It should not flatten the evaluation into one composite score without showing the underlying tradeoffs.
A product owner reviews the result. If the decision is consequential or the evaluator confidence is low, the company sends selected cases to a second evaluator and measures agreement.
The winning configuration still needs production monitoring. A 30-prompt test does not prove performance for every customer question.
What to measure
- blind preference rate;
- scores by dimension;
- critical errors;
- tie rate;
- evaluator confidence;
- inter-rater agreement;
- evaluation time and cost;
- later production escalation or correction rate.
State sample size and context whenever publishing the result.
Why the human adds value
The evaluator is useful because they can judge whether a person could act on the answer and whether its confidence matches its evidence. That is different from asking a model which output resembles an ideal response.
Human judgment also has limits. Evaluators can disagree, misunderstand the product, or prefer style. A fixed rubric, blind design, calibration examples, and a second evaluator reduce those risks.
What the company kept on record
The company retained the frozen prompts, output-generation settings, the anonymization method, the randomization record, the rubric, the accepted brief, the raw evaluations, the aggregation method, the disagreements, and its final decision.
No cherry-picked example is presented here as representative of the full test.