Agentic RAG beats modular rewriting but loses on routing and reranking

RAG is one of the most practical ways to connect an LLM to outside knowledge: instead of leaning on what it memorized, the model first pulls the relevant passages out of a knowledge base and only then answers. In shipped products it reads as the cure for hallucinations and stale facts. But plain RAG sometimes retrieves the wrong thing. So the industry took two roads.

The first is Enhanced RAG: a fixed pipeline assembled out of useful parts. A router decides whether retrieval is needed at all. A rewriter reformulates the query so it lines up better with the documents. A retriever pulls candidates. A reranker re-sorts them so the context is as precise as possible. Legible, deterministic, reproducible.

The second is Agentic RAG: instead of predefined modules, the LLM is handed the role of an agent. It decides for itself whether to search, whether the query is worth rewriting, whether to search again, whether it has enough context. It sounds appealing — fewer hard rules, more room to adapt. But is it actually better, and above all, under what conditions?

The paper takes on the uncomfortable question of what actually works, with a large-scale experimental comparison of Enhanced and Agentic RAG. And across different kinds of tasks: financial questions (FIQA), fact-checking (FEVER), finding similar discussions (CQADupStack-English) and open general-knowledge questions (NQ). The authors break RAG down into several bottlenecks and check who wins where.

Where RAG has to say "no search needed"

The first test is telling apart queries that genuinely need outside knowledge from queries that fall outside the knowledge base or are better answered without retrieval. In Enhanced RAG a separate embedding-based semantic router handles this: it compares the query against examples of valid and invalid queries and decides whether to fire retrieval. In Agentic RAG the LLM makes that call itself, as part of its reasoning.

On the financial and forum data the agent is genuinely good: it almost never misfires, and it even edges ahead of modular routing. But on FEVER — fact-checking claims about anything at all — the agent keeps retrieving where retrieval is pointless, and its detection quality falls off sharply. The lesson: the broader the domain and the blurrier the task boundary, the harder it is for an agent to reliably cut out what it doesn't need, and modular routing turns out to be the steadier option.

Rewriting the query, but not always the same way

The second test is query rewriting. Documents in the knowledge base tend to be long and formal, while the user's query is short and conversational. Rewriting helps the retriever close that gap. In Enhanced RAG the rewriting is forced and always uses one method — a HyDE approach, where the LLM writes a paragraph as if it were the answer and the search runs on that. In Agentic RAG rewriting is an optional step: the agent can reformulate as the situation and the document type call for it.

Here the agentic approach does pull relevant documents better on average. Flexibility really does seem to matter: sometimes a query is better expanded, sometimes better left alone, and sometimes better restyled for a particular corpus — forum threads, say. One universal, mandatory reformulation doesn't always land.

The third point is improving the document list after the initial retrieval. Enhanced RAG adds a classic cross-encoder reranker and picks the best items out of a large candidate pool. Agentic RAG could in principle reach something similar by running extra retrieval rounds with new phrasings. In the experiment it didn't: the agent's repeat passes don't produce meaningfully better documents, and sometimes make the picture worse.

So when the problem is that the top of the result list is full of junk, a reranker is often the more direct and more dependable fix than talking the LLM into rephrasing the query five more ways.

Dependence on the base LLM, and the unpleasant price of flexibility

The authors separately check whether the Enhanced vs Agentic balance shifts across base Qwen3 models (0.6B, 4B, 8B, 32B). The trend is the expected one: the better the LLM, the better the final answer. But the detail that matters is that the curves for Enhanced and Agentic rise in much the same shape — the agentic approach gets no magic bonus out of a stronger model.

In the measurements Agentic RAG consumes several times more input and output tokens (the multiple varies by task) and adds noticeably more latency. So even when the quality is comparable, you often pay more — simply because the agent reasons, chooses actions and takes extra steps.

What to take away if you're building RAG into a product

This work doesn't bury Agentic RAG, and it doesn't crown Enhanced as the one correct answer. It does something more useful: it shows where agency genuinely helps (flexible rewriting, narrow domains) and where modularity wins on robustness (routing on hard queries, dependable reranking). And it keeps the bill in view: agentic control almost inevitably means more tokens and more time, which means a more expensive answer.

Turned into engineering advice, the conclusion reads like this: agentic RAG is worth it where uncertainty is high and adapting on the fly pays off, but for the stable parts of the pipeline — reranking above all — the modular approach still looks stronger and cheaper.