A 31-subtype error taxonomy beats blind retries in text-to-SQL

Turning a human question into correct SQL is a surprisingly hard problem. Large language models write valid syntax well but miss the logic easily: they confuse tables, join on the wrong key, forget GROUP BY, apply the wrong filters. Plain self-correction from execution results doesn't always help — a query can run fine and even return something plausible that still isn't what the user asked for. The authors of SQL-of-Thought propose treating the cause rather than the symptoms: give the model a structure for its reasoning and a clear vocabulary of errors, so it isn't patching the query blind.

How SQL-of-Thought works

The idea is to split the path from question to SQL across several roles and make the models reason step by step.

The pipeline includes:

• schema linking: picking out the relevant tables, columns and relationships;
• decomposing the task into WHERE, JOIN, GROUP BY, ORDER BY and so on;
• a query plan in natural language: a chain of thought that explains what data is pulled and how;
• generating the SQL from that plan;
• guided correction: if the result doesn't match the reference, the system doesn't simply regenerate the query — it checks against the error taxonomy and builds a plan for the fix.

This isn't one clever prompt but a multi-agent system: different roles can run on different models, trading quality against cost.

Why an error taxonomy

The key novelty is an explicit classification of the model's mistakes. The authors extend an existing taxonomy to 9 categories and 31 subtypes, from schema and join problems to aggregations, subqueries and set operations. When a query returns the wrong answer, the correction agent doesn't guess — it states a diagnosis: "a foreign-key join is missing", say, or "an aggregation without the matching GROUP BY". From there it builds a short repair plan, and only then a new version of the SQL. That context teaches the model to fix the logic itself.

What the experiments show

The system was tested on the classic Spider and two harder variants — Realistic (explicit column names removed) and SYN (questions rewritten with synonyms). The headline metric is execution accuracy: whether the query's results match, not whether the SQL text matches exactly. The numbers are strong: 91.59% on Spider, 90.16% on Realistic, 82.01% on SYN — better than the current public systems on the same sets.

What the individual components contribute:

• A step-by-step plan before the SQL cuts errors: without it, accuracy fell by roughly 5%.
• Taxonomy-guided correction is what makes the system robust: switching that loop off cost up to 10% accuracy on the same data.
• Strong reasoning models (Claude Opus, for instance) matter most in the roles that have to understand the schema and plan. The mechanical SQL generation can run on cheaper models, which brings the price down with almost no loss in quality.

Practice and cost

A full Spider run on the strong model took about five hours and cost roughly $42–44 at temperature zero. A hybrid setup — the expensive model on schema and planning, a cheap one on the draft SQL and the final fix — saved a noticeable amount and reached about 85% accuracy on a trial sample. Yes, the multi-agent design means longer context and more API calls. Here that pays for itself in stability.

What works and what doesn't

• Works: think first, write SQL second. An explicit step-by-step plan disciplines the model and reduces hallucinations.
• Works: correcting with an explanation of which type of error occurred. Fewer useless retries that way.
• Didn't work: piping a long list of errors straight into the generation agent — better to update the plan first and patch the SQL only after that. Raising the temperature didn't help either, nor did running many independent "repairers": their edits conflict and add noise.

Where the weak spots are, and what comes next

For now the evaluation is confined to the Spider family, and the taxonomy hasn't been checked against every kind of enterprise schema. The system leans on closed reasoning models, which drives the price. The mature path looks like fine-tuning smaller models for specific roles: schema linking and correction strategy. The authors hint at exactly that plan — collect a corpus of typical errors and train a small assistant to diagnose and fix them quickly.

Why it matters

SQL-of-Thought makes a simple point carefully: structured reasoning plus legible feedback beats bare regeneration driven by execution signals. As access to data increasingly runs through natural language, we need systems that don't just write valid code but explain to themselves what they are actually doing — and why it needs fixing.