Top LLMs reason alike but diverge sharply on sycophancy and rephrasing

Today, evaluating a large language model comes down to a single number on a benchmark. That is convenient, and it is not enough: two models post identical scores and behave nothing alike in conversation. A group of researchers proposes looking deeper — taking a model's "behavioral fingerprint" along several axes to see how it actually thinks. The idea is simple: measure a profile of cognitive and interactive traits rather than one final score, from abstraction to sycophancy resistance.

Why one number is not enough

In the lab everything is tidy: accuracy on a test, a comparison, a conclusion. In practice we talk to an LLM, refine the wording, make corrections, ask about contested topics. That is where the things a classic benchmark misses come out: the urge to agree with the user, trouble with rephrasings, the manner of explaining, the confidence a model claims for itself. The authors introduce a multidimensional evaluation to make those differences visible and comparable.

What actually gets measured

The team assembled a set of 21 prompts that probe four areas:

• World model: whether models can reason through counterfactual physics ("What if…?");
• Abstraction and metacognition: generalization, analogy, confidence calibration;
• Personalization and bias: sycophancy resistance, communication style expressed through an MBTI-style test;
• Semantic robustness: consistency of answers across rephrasings of equivalent questions.

The scoring is automated: a strong LLM acts as an independent judge, comparing answers and returning a numeric score plus a short rationale. The results are then aggregated, normalized and drawn as radar charts.

How this was tested in practice

The authors ran 18 LLMs — nine frontier models and nine mid-tier ones. The set spans different architectures. The judge is Claude-opus-4.1, picked for how strictly it follows instructions. Each model gets a fingerprint that is easy to read: where its logic holds, where it drowns in rephrasings, where it gives in to the user.

What the numbers and the examples show

• Convergence in reasoning, divergence in behavior. The top models have roughly evened out on abstraction and causal chains. Other traits scatter widely. One axis is sycophancy resistance, another is semantic robustness.
• Counterfactual physics is still a problem. Even the leaders sometimes slide back to familiar facts about the real world instead of reasoning consistently inside the alternative universe they were handed. That marks the line between learned associations and genuine inference from first principles.
• Robustness to rephrasing varies. Given questions that are formally the same but worded differently, some models answer consistently; others shift their phrasing and emphasis more than you would want.
• The "personalities" share a common baseline. Profiles resembling ISTJ/ESTJ (ISTJ (Introversion, Sensing, Thinking, Judging) — the "inspector," the "logistician" / ESTJ (Extraversion, Sensing, Thinking, Judging) — the "executive," the "administrator") come up most often. Put another way, by default many LLMs answer in a structured, impersonal, decisive register. That may be a side effect of alignment through reinforcement learning from human feedback: the system is rewarded for clarity, restraint and logic.

What the fingerprint looks like on specific models

Individual radars lay out the balance of strengths and weaknesses. Some models, for instance, hold a steady line on abstraction and causality but lose robustness when the wording changes. Others are the reverse: they stand up to pressure from the user but are less stable across rephrasings.

Why this matters

• Picking a model becomes a real decision. With a behavioral profile instead of a single number, it is easier to match an LLM to the actual job: a helper for analytics, an assistant that talks to beginners, even a tool for generating scientific hypotheses in physics.
• Alignment is a design question. Similar cognitive ability does not guarantee similar behavior. Fine-tuning and alignment strategies shape the character of the interaction — sycophancy resistance, care in reporting uncertainty, conversational style.
• Risks surface earlier. Where a model tends to agree with the user, or loses consistency across rephrasings, you can harden the guardrails in advance, revisit the prompts, or change the answering policy.

Where the approach can go next

The prompt set is still small, and "personalities" are a convenient but crude vocabulary for describing style. What remains is to test the line between a default persona and simulated roles, and to watch over a longer horizon: how one LLM's behavioral fingerprint shifts as it is updated. One detail matters here — the method is reproducible and scalable, so it can be folded into regression tests and production monitoring.

💾 Code