Amazon's seller analytics agent skips SQL and answers in under 15 seconds

An e-commerce seller makes decisions on the fly all day: what to push in ads, where sales slipped, which products drag the business down and which drive growth. There is no shortage of data, but the value in it is rarely obvious. Getting an answer means opening several tools, knowing which report holds the number, how to build it, which filters to set — and then reading the result correctly. The authors of Insight Agents propose a more human route: let the seller "talk" to their own data in plain language and get back either a clean summary of the numbers or a business insight they can act on.

At the center is a conversational LLM system built as a multi-agent setup, where different roles handle different parts of the job. This is not a demo for the sake of a demo: the system was deployed for real Amazon sellers in the US and measured for accuracy and speed.

Inside the assistant: a manager and two specialists

Everything starts with the manager. Its job is to decide quickly whether the question can be answered from the available data at all, and what kind of answer is needed. The authors split user questions into two broad classes. In the first, the person wants to see data — sales and traffic for top products over the past month, say. In the second, they want diagnosis and explanation: what is happening to the business and why some metric may have slipped.

The manager does two things that matter a great deal in practice: it screens out questions that have come to the wrong place, and it routes the rest to the right branch. The authors also go after the standard affliction of LLM systems — latency. Instead of asking the LLM every time, they put lightweight ML modules wherever that is possible.

The first module is an out-of-distribution (OOD) detector. It exists so the expensive reasoning chain never fires on a question that isn't about analytics insights or isn't covered by the data. An autoencoder handles this, checking quickly whether a query looks like the admissible ones. The second module is the router: a compact BERT model decides whether the query goes to the data presentation side or to the insight generator. Both decisions are faster than LLM classification.

There is a third, more human piece — query clarification. A phrase like "last week" with no context turns into confusion easily. So the system writes the concrete time interval, and the rules for interpreting the period, straight into the prompt for the next step.

Why this isn't text-to-SQL, and where planning comes in

The most interesting engineering is in the handling of tabular data. Plenty of projects try to make the assistant write SQL. The authors go the other way: they build on internal data APIs and wrap them in an action-level data model. That narrows the freedom and sharply raises reliability — picking from a bounded set of correct operations is easier than composing a fresh database query each time and getting a column name or the syntax wrong.

Then plan-and-execute planning kicks in. The query is broken into subtasks, the right APIs and functions are selected, call parameters are generated (slot filling, in effect), and an executor carefully collects, aggregates and post-processes the tables. Only after all that does the LLM write the answer — either as a readable table or summary, or as an interpretation with domain hints.

The insight generator routes by domain scenario (performance, benchmarks, recommendations and so on). It works like rails for the reasoning: the LLM still writes the text, but along a path laid out in advance, with hints and examples supplied by domain experts.

What it does in production: accuracy and latency

The authors evaluate the system honestly, not only with classifier metrics but with human judgements of the answers. An important detail: they measure three concrete qualities rather than abstract "correctness" — how relevant the answer is to the question, how correct the insights are, and how complete the answer is.

What came out is a system that, in a live deployment, reached roughly 90% accuracy by human evaluation (question-level accuracy of 89.5% on in-scope questions) while holding P90 latency under 15 seconds (13.56s). For conversational analytics, where an answer usually means fetching data, aggregating it and only then explaining it, that is a strong engineering trade-off.

It is also telling that the fast ML modules genuinely earn their place: the OOD autoencoder gives very high precision and runs in a fraction of a millisecond per example, and routing on a lightweight BERT beats an LLM classifier on both accuracy and latency.

What this means and where it goes next

Insight Agents is a good example of how an LLM should be embedded in product analytics: not as the single brain for every case, but as one part of an orchestration that also holds fast filters, planning, constrained tools for reaching the data, and domain templates. That combination lowers the risk of hallucination, saves time and makes answers repeatable — which is what makes them usable by a business rather than only in a demo.

The authors name the next step: widen scenario coverage and build out automatic quality evaluation. That follows: the more real questions the system sees, the more it matters to be able to check it quickly without constant manual labeling.