The AI judge maintains 99% accuracy at a lower cost.

A Cheap Judge for AI

Evaluating AI responses is expensive—especially when you need to check not hundreds of examples, but millions of responses in a data pipeline, benchmark, or data-selection system.

The usual approach is to use a powerful language model. It reads the question, compares the responses, checks the facts, and delivers a verdict. The problem is cost and latency. The more capable the model and the longer its reasoning, the more each call costs.

A Carnegie Mellon paper proposes a different approach: use a cheap judge first, and call on a stronger model only when the first judge is uncertain. The first judge is JEV—a service that returns not an explanation, but a typed verdict and probabilities for the possible labels.

On standard tasks, JEV came close to GPT-6. At the same time, each of its verdicts cost about 277 times less.

What Is JEV?

A generative judge typically writes out its reasoning and then provides a final answer. JEV works differently. It receives a question, a description of the labels, and the input data. It returns a structured response containing:

🟠 the selected label;

🟠 the probabilities of all possible labels;

🟠 a confidence score that can be used for routing.

For example, a response-comparison task might use the labels “response A” and “response B.” A fact-checking task might use “supported” and “hallucination.” For checking a final answer, the labels could be “correct,” “incorrect,” and “no answer.”

This interface works well in a pipeline. There is no need to parse a long explanation and try to determine exactly what conclusion the model reached. The system receives a value of the required type immediately.

Comparison of human evaluation, a generative judge, and a judge with a typed verdict and probabilities.

But the approach comes with one condition: the probability must help distinguish correct decisions from mistakes. If the model is confidently wrong, routing will not save you.

What Was Tested

The researchers compared JEV with 16 judges, including generative models, local models, and reward models. The comparison included GPT-6, which served as the main quality benchmark.

The evaluation covered several types of tasks:

🟣 comparing two answers to the same question;

🟣 checking an answer against provided evidence;

🟣 evaluating a model’s final answer against a reference answer;

🟣 checking challenging tasks involving knowledge, mathematics, reasoning, and programming;

🟣 selecting the best answer from four options;

🟣 comparing answers written in different styles.

The main datasets came from RewardBench, JudgeBench, and HaluEval. The authors also used saved responses from experiments with multi-turn dialogues, along with several dedicated control sets.

The basic comparison included 1,312 tasks. Format errors also counted as errors. If a model failed to return a valid label, violated the schema, or failed to respond after repeated attempts, the result was not corrected manually.

To examine disputed cases, the authors conducted a blind human evaluation. The expert could not see the benchmark labels and did not know which judge had been correct.

JEV Handles Routine Tasks Cheaply

On RewardBench, JEV scored 92.2%, compared with 93.5% for GPT-6—a difference of 1.3 percentage points.

On HaluEval, where an answer must be checked against evidence, JEV scored 87.5%, while GPT-6 scored 86.7%. The result favored JEV, although both the sample and the labels themselves included disputed cases.

On a set of 150 saved responses, JEV reached 94.0%, compared with 96.7% for GPT-6.

In other words, on routine comparison tasks and answer checking against ready-made context, JEV stays close to a far more expensive model.

🟠 RewardBench: 92.2% for JEV versus 93.5% for GPT-6.

🟠 HaluEval: 87.5% versus 86.7%.

🟠 Final-answer evaluation: 94.0% versus 96.7%.

🟠 Cost of JEV: approximately $0.044 per thousand decisions.

🟠 Cost of GPT-6 on the same evaluation set: approximately $12.182.

Comparison of latency and the cost of one thousand decisions for hosted models.

The difference in speed is also significant. JEV’s median latency was 0.152 seconds, compared with 1.885 seconds for GPT-6. This measurement includes the network, the provider’s server, and request retries, so it should not be treated as pure inference speed. But for an applied pipeline, the total time to obtain a result is what matters.

The Gap Widens on Difficult Tasks

JudgeBench produced a different picture. These tasks require evaluating whether an answer is objectively correct, including in mathematics, reasoning, and programming.

JEV scored 78.6%, while GPT-6 scored 93.1%—a difference of 14.6 percentage points.

The gap was especially pronounced in reasoning and programming:

🟣 reasoning: 68.4% for JEV versus 95.9% for GPT-6;

🟣 programming: 76.2% versus 97.6%;

🟣 knowledge: 84.4% versus 90.9%.

The human evaluator who examined the disputed cases mostly agreed with GPT-6. On JudgeBench, the expert chose GPT-6 in 57 of the 69 cases where the judges disagreed. JEV was selected only once.

JEV struggles when it has to independently verify a chain of reasoning. A quick assessment of the answer is not enough. The judge must reconstruct the solution, find an error in a formula, or recognize that a persuasive-looking answer ends with the wrong conclusion.

A similar pattern appeared on RM-Bench. When the correct answer was written more simply and the incorrect answer was more detailed and polished, JEV was more likely to choose the wrong option. With matching styles, its accuracy was 84.0%. With difficult style pairs, it fell to 74.8%. For GPT-6, the effect was almost nonexistent: 93.3% and 94.6%, respectively.

Accuracy of JEV, GPT-6, and the reward model on tasks involving the selection and comparison of answers written in different styles.

Confidence Becomes a Router

The paper’s practical idea is not to use JEV on its own.

The setup works as follows:

1. JEV quickly evaluates each example.

2. If the maximum probability is high, the system accepts its verdict.

3. If confidence falls below a threshold, the example is passed to GPT-6 or another powerful model.

4. The second judge’s decision becomes the final result.

This is a standard cascade, but its effectiveness depends on whether the first judge can identify its own mistakes.

JEV does show such a relationship. Across the three main datasets, accuracy increased along with the maximum probability assigned to the selected label. Among decisions with a probability of 0.99 or higher, it rarely made mistakes. Among decisions with a probability below 0.6, only 47.7% were correct.

Distribution of JEV’s confidence on correct and incorrect decisions, along with the reliability curves of different judges.

In a cascade simulation with a threshold of 0.9, 34% of examples were sent to the stronger model. The cascade retained 99.6% of GPT-6’s accuracy while requiring about 47% of its cost. On JudgeBench, more tasks had to be escalated—about 61% at the same threshold. Even there, the cascade retained 98.2% accuracy and cost roughly 62% as much as GPT-6.

In a preregistered experiment on pairwise comparisons, JEV handled 53.7% of decisions itself. The rest were sent to GPT-6. Accuracy was 92.5%, compared with 93.1% for GPT-6 without a cascade. Relative cost was about 57%.

The cascade accepts JEV’s confident decisions and passes uncertain cases to a stronger model.

Confidence Is No Guarantee

Routing has its limits.

On difficult style pairs, JEV sometimes confidently selected the wrong answer. The situation was even worse on tasks that required checking free-form text without evidence. JEV scored 52.5%—almost random. Yet its average maximum probability was around 0.90.

That creates a dangerous combination: the judge barely understands the task but reports high confidence. In this mode, a threshold does not help. The system will accept mistakes as reliable decisions.

The authors also tested whether calibration transferred between datasets. A temperature calibrated on one type of task sometimes improved probabilities on another, but it could also make them worse. No universal value emerged.

For practical use, local checks are essential:

🟣 check both candidate orders in pairwise tasks;

🟣 choose the threshold on a separate holdout set;

🟣 count format errors as full-fledged errors;

🟣 test tasks involving misleading styles separately;

🟣 do not transfer a threshold or calibration to a new type of data without measuring it again.

What This Changes for Evaluation Pipelines

JEV does not replace a powerful AI judge. It reduces the number of times you need to call one.

For large-scale evaluation of preferences, answers based on provided sources, and the straightforward selection of final responses, a cheap first pass makes sense. The system can process most of the data quickly and send only disputed cases to the expensive model.

For mathematics, programming, multi-step reasoning, and answers that must be checked without an external source, JEV alone is not enough. These tasks call for a stronger model from the start or a dedicated verification process.

Relationship between JEV’s errors and confidence, and the cascade’s savings when uncertain decisions are passed to GPT-6.

There is also a broader lesson. Verdict, correctness, and reliable confidence are three different properties. A model can follow the required format, choose the correct label most of the time, and still be poor at recognizing when it is wrong.

Conclusion

JEV works well as a cheap first judge. On routine tasks, it trails GPT-6 by a few percentage points, but costs orders of magnitude less and responds faster.

The “accept confident decisions, escalate uncertain ones” cascade can preserve about 99% of GPT-6’s quality at roughly half the cost. But it works only within the type of task that has been tested.

If you are building an AI evaluation pipeline, a sensible setup looks like this:

🟠 a fast judge for most examples;

🟠 probability as a routing signal;

🟠 a powerful model for difficult and uncertain cases;

🟠 separate threshold testing on your own data.

Model confidence should be treated as a reason to perform an additional check—not as proof that the answer is correct.