November 9, 2026

LLM-as-a-judge: building an AI grader you can actually trust

An LLM judge works when calibrated against human labels — without calibration it's noise dressed up as a metric. Here's how to build one that holds up in CI.

Short answer: An LLM judge works when you’ve calibrated it against human labels. Without calibration it’s noise dressed up as a metric. The rubric is the actual work — the judge model just executes it.

Teams reach for LLM-as-judge because there’s no other way to automatically score whether a support reply was “empathetic” or whether a generated SQL query “answers the user’s intent.” There’s no diff to run. The judge fills the gap between “does it parse” (easy to check mechanically) and “is it actually good” (hard to check without a human).

The failure mode is treating it as plug-and-play: pick a model, write a prompt like “rate this output from 1–10,” drop it in CI, declare victory. That judge gives you a consistent number that correlates poorly with what users actually think. You’ve built a metric that feels like signal and isn’t.

Here’s how to build one that isn’t that.

Why do LLM judges fail without calibration?

Three problems, all fixable.

Position bias. In pairwise comparisons, judges prefer whichever answer appears first — consistently enough to flip verdicts up to 30% of the time. A test that scores “response A vs response B” will sometimes return a different result just from swapping the order.

Verbosity preference. Most LLMs, when asked to judge quality without a tight rubric, prefer longer outputs. A wrong-but-detailed answer beats a correct-but-brief one. Your eval suite ends up rewarding word count. Users don’t.

Same-model blindness. Claude evaluating Claude’s outputs is lenient on its own artifacts. It misses the specific failure modes it produces. If your app uses a model and you judge with the same model, you get a blind spot for exactly the class of errors you need to catch.

What makes a rubric tight enough to trust?

The rubric is not the prompt template. It’s the list of concrete criteria the judge checks against. Write it as conditions, not adjectives.

Bad rubric:

Score 1–10 for quality. Consider clarity, accuracy, and helpfulness.

Good rubric:

A good response:
- Directly answers the question asked (not a related question)
- Does not introduce information not present in the provided context
- Uses plain language; no jargon unless the user used it first
- Is under 200 words

Score 0.0 immediately if any of:
- Response says "I don't know" when the context contains the answer
- Response asks a clarifying question instead of answering
- Response repeats the system prompt

Otherwise score 0.0–1.0 against the criteria above.

The hard-failure conditions matter most. They cover cases where you know it’s wrong — no rubric ambiguity. Put them first. They short-circuit scoring before interpretation enters.

Here’s a complete judge implementation:

import anthropic

RUBRIC = """\
A good response to a customer support question:
- Directly addresses the specific question asked
- Does not introduce facts not present in the provided knowledge base context
- Is under 200 words
- Uses plain language; no jargon unless the user introduced it
- Ends with a concrete next step or resolution

Score 0.0 immediately if any of the following:
- Says "I cannot help with that" when context contains the answer
- Asks a clarifying question instead of answering
- Contradicts information in the context
- Contains a system prompt fragment

Otherwise score 0.0–1.0 based on the criteria above.
Return only a decimal number (e.g., 0.85). No explanation."""

client = anthropic.Anthropic()

def judge_response(user_question: str, kb_context: str, response: str) -> float:
    prompt = f"""User question: {user_question}

Knowledge base context:
{kb_context}

Response to evaluate:
{response}

Rubric:
{RUBRIC}"""

    result = client.messages.create(
        model="claude-haiku-4-5-20251001",
        max_tokens=10,
        temperature=0,                 # deterministic — same input, same score
        messages=[{"role": "user", "content": prompt}],
    )
    try:
        return float(result.content[0].text.strip())
    except ValueError:
        return 0.0                     # unparseable output = treat as failure

Two things to notice: temperature=0 for determinism, and the parsing fallback returns 0.0 instead of crashing. A judge that sometimes returns “0.85 (good answer)” instead of “0.85” should fail safe, not silently inflate your scores.

How do you calibrate before shipping to CI?

You need 20–30 examples labeled by a human against your rubric. Not labeled “correct” — labeled against each criterion: does this response satisfy the condition or not?

# calibration.jsonl — one entry per labeled example
# {"question": "...", "context": "...", "response": "...", "human_score": 0.9, "notes": ""}
# {"question": "...", "context": "...", "response": "...", "human_score": 0.1, "notes": "contradicts context in para 2"}

import json

def calibrate_judge(labeled_path: str, threshold: float = 0.2) -> dict:
    with open(labeled_path) as f:
        examples = [json.loads(line) for line in f if line.strip()]

    disagreements = []
    for ex in examples:
        judge_score = judge_response(ex["question"], ex["context"], ex["response"])
        delta = abs(judge_score - ex["human_score"])
        if delta > threshold:
            disagreements.append({
                "question": ex["question"][:80],
                "human": ex["human_score"],
                "judge": judge_score,
                "delta": round(delta, 2),
                "notes": ex.get("notes", ""),
            })

    agreement_rate = 1 - len(disagreements) / len(examples)
    return {
        "agreement_rate": round(agreement_rate, 3),
        "disagreements": disagreements,
        "verdict": "PASS" if agreement_rate >= 0.80 else "RUBRIC NEEDS WORK",
    }

Run this before wiring the judge into CI. If verdict is RUBRIC NEEDS WORK, look at the disagreement list. Almost always, it’s not that the judge model is wrong — it’s that your rubric doesn’t say what you thought it said. Add the failing condition explicitly and re-run.

Eighty percent agreement is the floor, not the goal. Below 80%, the judge is unreliable and your CI metrics will mislead you into thinking quality is stable when it isn’t.

How do you handle position bias in pairwise comparisons?

Run each pair twice with the order swapped. If the judge flips its verdict, the difference isn’t real — call it a tie.

def pairwise_judge(question: str, response_a: str, response_b: str) -> str:
    """Returns 'A', 'B', or 'TIE'."""

    def score_pair(first: str, second: str) -> str:
        result = client.messages.create(
            model="claude-haiku-4-5-20251001",
            max_tokens=5,
            temperature=0,
            messages=[{"role": "user", "content": (
                f"Question: {question}\n\n"
                f"Response 1: {first}\n\n"
                f"Response 2: {second}\n\n"
                "Which response better satisfies the rubric above? "
                "Reply only: 1, 2, or TIE."
            )}],
        )
        return result.content[0].text.strip()

    verdict_ab = score_pair(response_a, response_b)    # A presented as 1, B as 2
    verdict_ba = score_pair(response_b, response_a)    # B presented as 1, A as 2

    # normalize to A/B labels
    winner_ab = "A" if verdict_ab == "1" else ("B" if verdict_ab == "2" else "TIE")
    winner_ba = "A" if verdict_ba == "2" else ("B" if verdict_ba == "1" else "TIE")

    if winner_ab == winner_ba:
        return winner_ab
    return "TIE"   # order-sensitive result = not a real difference

For single-answer scoring — which is what you want in most CI setups — position bias doesn’t apply. There’s only one answer, and you score it against the rubric directly.

What does LLM-as-judge not solve?

It doesn’t tell you if the rubric is right. If you’re measuring the wrong thing — if “under 200 words” is in your rubric but users actually want detailed answers — calibration passes and your eval suite silently optimizes for the wrong outcome. No amount of judge engineering fixes that. Review your labeled examples with someone who understands the product intent before locking the rubric.

It also shouldn’t replace mechanical verification. If your LLM writes SQL, run it against a test database. If it writes code, execute it. Use a judge for genuinely subjective dimensions only — the ones where there’s no ground truth to diff against. Applying a judge to things you can verify deterministically wastes money and adds noise.

The free test lab has a complete judge harness with calibration script, a 25-example labeled dataset for a support-reply task, and a CI integration that fails the build when judge agreement drops below 80% after a model upgrade. If you’re setting this up for the first time, start there rather than from scratch. AI Software Testing in Practice covers the full eval lifecycle — growing the labeled dataset across model upgrades, setting regression thresholds that account for judge noise, and knowing when a score drop is a real problem versus rubric drift.

FAQ

How do I know if my LLM judge is reliable? Run it against 20–30 human-labeled examples and measure agreement. If the judge disagrees with your labels more than 20% of the time, the rubric is wrong — tighten it and re-calibrate. Agreement above 80% on your labeled set is enough to trust the judge in CI.

Should I use the same LLM to generate and judge outputs? No. Using the same model to evaluate its own outputs creates a blind spot for that model’s specific failure modes. Use a different judge model, or judge with a stronger model than the one generating outputs.

What is position bias in LLM-as-judge and how do I avoid it? Position bias means the judge tends to prefer whichever option it sees first in a pairwise comparison. Avoid it by either using single-answer scoring (not pairwise) or running each comparison twice with the order swapped and averaging the scores.

← All guides