LLM as a Judge for Evaluating Agent Responses: A Practical Method

LLM as a Judge for Evaluating Agent Responses: A Practical Method

LLM as a judge means using a language model to score or rank the responses of another model or agent against a written rubric. It works well for criteria that simple rules cannot capture, such as tone, completeness and faithfulness to context. It only works if the judge is calibrated against samples reviewed by people, and it should never decide on its own anything that carries risk.

Key takeaways

• An automated judge widens evaluation coverage. It does not replace human review. People define what a good response is, and the model applies that standard at volume.
• An uncalibrated judge is an opinion dressed up as a metric. Measure agreement with human reviewers before the score becomes an indicator.
• The biases are predictable: a preference for longer responses, for the first option shown, and for the model's own style. Each one can be tested in your own case.
• Objective criteria (the agent called the right tool, the value matches the system of record) should be checked by code, not by a model.
• Calibration is ongoing. The human sample comes back periodically to catch drift in the judge, the agent or the kind of questions customers ask.

What LLM as a judge is and what it evaluates

In practice, the judge receives four things: the customer's question, the agent's response, the context the agent had (company policy, order data, a document excerpt) and a rubric. It returns a score or a verdict per criterion, with a short justification. That output is stored alongside the conversation record.

The value is in evaluating what is hard to write as a rule. A response can be correct and still too curt for an irritated customer. It can cite a policy that does not exist. It can promise a deadline nobody authorized. Reading conversation by conversation does not scale, and plain text rules miss these cases.

What the judge should not evaluate alone: facts that live in a system. If the agent said the order shipped, code queries the ERP and checks. If the agent should have handed off to a person, the log shows whether the handoff happened. Using a model for something code can verify adds cost and error with no gain.

Anthropic's documentation on evaluating agents and OpenAI's evals guide treat this split in a similar way: combine deterministic checks with model-based graders and human review, each where it fits best.

When it makes sense to use one model to grade another

Use a judge when the criterion is qualitative, the volume of conversations is more than the team can read, and there is a written definition of what counts as acceptable. Without that definition, even people disagree with each other, and the model will simply inherit the confusion.

Signs the judge will help:

• The criterion depends on interpretation: appropriate tone, clarity, faithfulness to a document, adherence to a policy.
• Volume is high enough that reading everything is impractical, but a small sample can be reviewed by people.
• You need to compare versions of the agent (new prompt, new model, new knowledge base) before publishing the change.
• The team can explain, in plain sentences, what separates a good response from a bad one.

Signs it will not help, or will help only a little:

• The criterion can be verified by code or a system lookup. In that case, write the test.
• The decision has high consequences, such as billing, cancellation, diagnosis or a contractual promise. Human review is mandatory here, and the judge at most helps prioritize what to review.
• There is no sample labeled by people. Without one, there is no way to know whether the judge's score means anything.
• The team expects the judge's score to be the final truth. It is an estimate, with a margin of error that needs to be known.

When choosing which model to use as the judge, remember that it does not have to be the same one as the agent, and often should not be (the bias section below explains why). A practical comparison of models on a set of real tasks, in the spirit of Kimi K2.5 vs Claude Opus 4.5, also works for picking the judge: test on your own cases, not on third-party tables.

Which biases a judge has and how to test for them

The biases below are often described in technical material on model-based evaluation. Treat them as hypotheses to verify in your operation, not as facts about every model and every task. The good news is that each one has a cheap test.

• Length bias: the judge tends to favor longer, more detailed responses, even when the short one solves the problem better. Test: rewrite good responses more tightly and see whether the score drops for no good reason.
• Position bias: in comparisons between two responses, the one shown first may win more often. Test: swap the order and compare results. If the verdict changes, the comparison is not reliable.
• Self-preference: a model may grade text that resembles what it would write itself more favorably. Test: use a judge from a different model family than the agent and compare scores against the human sample.
• Authority and format bias: responses with a confident tone, lists and technical terms can look better than they are. Test: include well-formatted but wrong responses in the sample and see whether the judge penalizes them.
• Leniency: the judge often gives medium responses scores that are too high, which hides regressions. Test: check whether the score distribution uses the whole scale or clusters at the top.
• Sensitivity to rubric wording: small changes to the judge prompt shift the scores. Test: run the same sample with two equivalent wordings and measure the difference.

There is also a security risk. The judge reads text written by customers and by the agent, and that text can contain disguised instructions, such as a request to give the top score. Separate clearly, in the prompt, what is rubric and what is content under evaluation, and treat the content as data, not as orders. OWASP's guidance on preventing prompt injection describes mitigation practices that apply to this case too.

How to calibrate the judge with human-reviewed samples

Calibrating means measuring how often the judge agrees with people who know the business, then adjusting the rubric until agreement is acceptable for the intended use. A step-by-step approach that works for most operations:

• Step 1, define criteria: write three to five independent criteria, each with one objective question. Example: does the response use only information present in the context? Is the customer's next step clear? Did the agent hand off when policy required it?
• Step 2, build the sample: collect real conversations, including easy cases, hard cases, complaints and situations where the agent should have refused or handed off. Do not pick only the good ones. Before any review, remove or mask personal data that is not needed, in line with the data minimization principle in Brazil's LGPD.
• Step 3, label with two people: two reviewers score the same sample with the same rubric, without seeing the judge's score. Where they disagree with each other, discuss and adjust the rubric. If humans do not agree, the criterion is poorly defined.
• Step 4, run the judge: apply the judge to the same sample and compare it with the human consensus, criterion by criterion. Look at the errors, not just the average: which cases it approves that people reject, and the reverse.
• Step 5, fix and repeat: adjust the rubric, add examples of low and high scores, ask for the justification before the score, and run again. Keep part of the sample out of the tuning, so you do not calibrate only for cases you have already seen.
• Step 6, define permitted use: once agreement is measured, decide what the score may do. For example, comparing agent versions and prioritizing conversations for review are fine. Blocking a response or penalizing an employee is not.
• Step 7, recalibrate periodically: draw new conversations for human review at a fixed cadence. If agreement drops, something changed: the agent, the audience, the products or the judge itself.

When measuring agreement, be suspicious of a high rate if almost everything in the sample is easy. The sample must contain borderline cases, because that is where the judge errs and where the business loses.

Illustrative scenario: a scheduling agent on WhatsApp

Imagine, as an example, a clinic that uses a WhatsApp agent to book, reschedule and cancel appointments, as described in automatic appointment scheduling on WhatsApp with AI. The team wants to know whether a prompt change improved or worsened the service.

First, they separate what code can check: the confirmed slot exists in the calendar, the patient and the practitioner are correct, no slot was double-booked. This becomes an automated test, no model involved.

Then they list what depends on judgment: the response was courteous to someone in a hurry, the agent avoided clinical advice that is not its place to give, the refusal was clear when the request fell outside scope, and the handoff to the front desk happened when the patient insisted or showed discomfort. Those four criteria go into the judge's rubric.

The front desk reviews a sample of real, masked conversations and compares them with the judge's scores. They find that the judge approves responses that are too long and is lenient with advice that sounds clinical. The team adds examples of improper advice to the rubric, asks the judge to quote the problematic passage, and switches to a model from a different family than the agent's. They repeat the comparison. Only then is the score used to compare prompt versions, always with a set of borderline conversations reviewed by people before anything is published.

Note what the score does not do in this example: it does not cancel appointments, penalize anyone or replace the human handoff flow. It guides where to look.

How to write the rubric and the judge prompt

Judge quality depends more on the rubric than on the model. A few practices that reduce noise:

• One criterion per question. Asking for a single score for overall quality mixes different things and hides the cause of a problem.
• Short scales with definitions. A verdict of pass, fail or partial, with a description of each, is usually more stable than a long, vague scale.
• Real examples of each level, including borderline cases, taken from the sample reviewed by people.
• Justification before the verdict, so the score follows from an analysis rather than a guess.
• Full context. The judge can only assess faithfulness if it sees the same documents and data the agent saw.
• Structured output, so the score can be stored, aggregated and audited.
• Permission to say it does not know. A judge forced to always decide invents certainty.

Store the rubric version and the judge model version with every score. When a metric changes suddenly, you need to know whether the agent changed or the measuring instrument did.

How this fits the method: map, limit, pilot, hand over

An automated judge is one part of the pilot, not the whole pilot. In the method we use, the first step is to map the current process and the baseline: how quality is evaluated today, who reviews, which errors cost the most. Next comes limiting: defining what data the judge may see, what actions the score may trigger (ideally none automatic at the start) and when human review is mandatory.

In the pilot, you choose one real flow, with records and an agreed success metric, and the judge enters already calibrated against the human sample. To set those criteria explicitly, the AI pilot charter template and the article on how to measure an AI automation pilot help you record what will be measured and when to stop. At handover, the company receives the rubric, the labeled sample, the recalibration routine and the rule for who responds when agreement drops, so it can decide whether to expand, adjust or end the project.

Frequently asked questions

Can an LLM judge replace human review? No. It widens coverage and lets you compare versions quickly, but the quality reference remains the judgment of people who know the business. Sampled human review should continue, especially for risky cases.

Does the judge need to be a stronger model than the agent? Not necessarily. What matters is the agreement you measure with human reviewers. A judge from a different model family than the agent helps reduce self-preference. Test more than one option on the same reviewed sample and choose by result, also considering the cost of running at volume.

How much agreement with humans is enough? It depends on the use. To prioritize conversations for review or compare versions, moderate agreement may be enough, as long as the errors do not cluster in the serious cases. For any automatic decision that affects a customer, the judge alone is not enough, and the decision should go through human approval.

Can I send real customer conversations to a model for evaluation? Only with care. Conversations contain personal data, so apply minimization, mask what is not needed, have an adequate legal basis and keep a record of the processing, as LGPD requires. Also check where the model provider processes and stores the data, and consult legal counsel or your data protection officer before expanding the use.

Sources

• Anthropic: Demystifying evals for AI agents
• OpenAI: Evals guide
• OWASP: LLM Prompt Injection Prevention Cheat Sheet
• NIST: AI Risk Management Framework