The best AI agent scores only 66% when data is spread across files

When one table isn't enough for an AI agent

Picture an ordinary work request: compute a fund's risk, pull the right equity records, assemble a medical summary. In practice the answer almost never sits in one clean table. Part of it hides in SQLite, part in a long PDF, some of the criteria are spoken aloud in a video, and the question itself may be in a different language. For a person this is tedious. For an AI agent it is a routine failure.

DataSpace is a new benchmark for AI agents that have to work across a heterogeneous workspace rather than a toy "write one SQL query" task. The authors assembled 410 tasks, 7,439 files and roughly 15 GB of data, and made the agent solve the whole problem end to end: find the right sources, extract the facts, match the entities, compute the metrics and return a finished table.

The headline result: even the best current systems are far from working confidently in this mode. Top accuracy is 66.34%. The leader gets roughly one task in three wrong.

This matters if you are building AI agents for analytics, internal company data, financial workflows or data-pipeline automation. On tidy demos everything looks convincing. In a real workspace it stops looking that way.

🟠 DataSpace tests the full analytical loop

🟣 Best result: 66.34%

🟠 Roughly one task in three comes back wrong

🟣 The core scenario is messy local data, not a single SQL query

What DataSpace is

In DataSpace an AI agent gets only a natural-language question and a folder holding a local workspace. Inside there may be CSV, JSON, SQLite, Markdown, PDF and video. The agent's job is to return the complete final table, which can then be checked automatically.

That sets it apart from many older benchmarks. Previously one test checked SQL, another document search, a third video question answering. Here all of it is fused into a single scenario.

The task conditions are closer to reality too:

🟠 question and data may be in different languages, including Chinese and English

🟣 the relevant artifacts are not marked in advance — the agent has to work out where to look

🟠 the answer counts only as a complete table

🟣 grading is deterministic: either the table matches in substance or it does not

The authors call such an agent a "workspace solver." It has to do more than extract a fact — it has to make sense of its environment.

The scale, briefly

🟠 410 tasks

🟣 7,439 artifacts

🟠 15.01 GB of data

🟣 6 media types: CSV, JSON, SQLite, Markdown, PDF, video

🟠 265 cross-lingual tasks

🟣 126,409 rows across the reference answers

Why the old benchmarks no longer suffice

DataSpace has a practical complaint about the earlier tests. They usually probe one isolated ability instead of the full analytical loop.

If you test only Text-to-SQL, the model works in a nearly sterile setting: the schema is given, the relevant database is known, one query gets the answer. If you test only long documents, you are mostly measuring search and extraction. If you ask for an open-ended analytical report, the grading tends to be fuzzy.

In a real business scenario the problem is harder:

🟠 the facts you need are scattered across different file types

🟣 some of the data has to be found first and then matched up

🟠 documents and video may hold not the answer but only the filter conditions

🟣 the user expects a finished table, not free text

So DataSpace fixes a single contract: the output must always be a complete table, and the evaluation must be formal, with no model acting as judge.

How the benchmark was built

The most interesting technical part is not the tasks themselves but how they were made. The authors did not build everything by hand from scratch. Instead they took executable Text-to-SQL sets from medicine and finance and turned them into dirtier, more realistic workspaces.

The pipeline is called DataSpace-Builder. It has four steps.

How the build pipeline works

First, the source tasks were moved into a cross-lingual setting. Not just the question was translated, but the database itself, the values in the columns and the SQL logic. That detail matters: translate the question and the database separately and you can break the links between entities, values and conditions.

Next the authors sampled the data locally for each task, so the same databases would not turn up in dozens of near-identical examples. But naive random sampling is dangerous here: you can drop a row needed to join two tables, or delete a value a filter depends on. So they added guard constraints to keep the critical links and conditions from vanishing.

Then the data was routed across media. Some tables stayed in CSV and JSON, others were packed into SQLite, others turned into long Markdown and PDF documents. A separate branch produced video, which could end up carrying the task's conditions or parts of the answer.

Finally, every task went through human review. Eleven domain experts took part. They solved each task independently, checked the reference answer, tuned the grading rules and fixed the problem cases.

What the pipeline consists of

🟠 cross-lingual transformation of the question, the database and the SQL

🟣 row sampling that preserves the links that matter

🟠 routing across media: tables, documents, SQLite, video

🟣 manual review and correction by experts

What DataSpace actually tests

The benchmark loads several abilities at once that look solved when you take them one at a time.

First, search across the workspace. The agent has to work out which files it even needs.

Second, extraction and matching. It has to pull the right fields out of a document, recognize that the same entity can go by different names, and not confuse units or languages.

Third, the analytics proper: filtering, aggregation, sorting, joins, time conditions.

Fourth, materializing the answer. This part is underrated. Even when the agent understands the task correctly, it can hand back the wrong shape of table: drop a column, add one that does not belong, scramble the row order, break a date or percentage format.

By the tasks' own statistics:

🟠 filtering is required in 78.3% of tasks

🟣 column projection in 82.4%

🟠 sorting in 51.0%

🟣 aggregation in 35.6%

🟠 joins in 27.6%

🟣 document understanding in 135 tasks

🟠 video understanding in 97 tasks

The telling number is that 60.5% of tasks demand at least five non-trivial operation types at once. That is already a small analytical chain.

🟠 DataSpace tests a chain of steps, not one skill

🟣 The common failure is the shape of the final table, not the computation

🟠 Most tasks need several operation types at once

What the experiments showed

The authors tested six frontier multimodal models and five agent scaffolds. In the first set they held the agent logic fixed and swapped only the base model. In the second they did the opposite: fixed MiMo-V2.5 and swapped the scaffold.

The main fact: nobody closed this benchmark.

Results by model

🟠 Grok 4.5 — 66,34%

🟣 GPT-5.6 Sol — 64,63%

🟠 Kimi K3 — 53,41%

🟣 MiMo-V2.5 — 39,27%

🟠 Claude Sonnet 5 — 32,93%

🟣 MiniMax M3 — 28,54%

The spread is wide, but the more important point is that even the leader solves only two thirds of the tasks.

There is a second result. GPT-5.6 Sol comes close to Grok 4.5 on accuracy while spending noticeably less: 74.2% fewer tokens, 50.3% fewer actions and 39.2% less time. A comparable result can be reached far more compactly.

Results by agent scaffold

With the model held fixed, the spread between scaffolds came to 15.36 percentage points. That is a lot.

Put plainly: what matters is not only which LLM you have, but how the agent itself is built — its system prompt, its tools, its context management, its step logic, the way it decides it is finished.

For applied development that reads almost as an instruction: improve the scaffold around the model, not only the model.

🟠 Leader: Grok 4.5 at 66.34%

🟣 GPT-5.6 Sol gives up almost nothing in quality and costs noticeably less to run

🟠 Gap between scaffolds: 15.36 points

🟣 Agent design moves the result nearly as much as the base model

Where the systems break

The most useful part is the error analysis. The authors looked separately at 136 failures from Grok 4.5, the best model in the controlled comparison.

The picture turned out to be unexpected. The main problem is neither finding the source nor the computation.

Most often the agent loses the task at the last step:

🟠 52.2% of the errors were in materializing the answer

🟣 in 44.1% of all analyzed failures the agent added or dropped columns

🟠 another 22.8% came from misreading which result the task was asking for

🟣 only 3 of the 136 cases involved picking an entirely wrong data source

The obvious barrier for an AI agent looks like finding the right file. What shows up here is something else: even with the right data in hand, and sometimes the right intermediate result computed, the agent cannot reliably deliver the exact final table.

Translated into product terms: understanding the meaning is not enough. The agent also has to hold strictly to the answer contract.

🟠 The dominant error type is answer materialization

🟣 Dropped or added columns: 44.1% of the analyzed failures

🟠 Picking the wrong data source is rare

Which tasks are hardest

The authors also broke the results down by task type. Two factors degrade quality for nearly every model.

The first is multimodality. When several media types have to be combined, accuracy drops for every model by 1.8 to 14 percentage points.

The second is table joins. When a task requires a join, quality falls by 9.7 to 19.8 points.

The sore spot is not a PDF on its own or a video on its own, but assembling an answer out of several sources at once.

What hurts quality most

🟠 combining several modalities

🟣 joins across tables and entities

🟠 cross-lingual matching

🟣 assembling the final table exactly

Answer size on its own does not explain everything, though. Answers with many rows or many columns are not necessarily worse. What matters far more is the path you have to take to get there.

Why this matters in practice

DataSpace is useful as more than one more academic task set. It tests precisely the scenario companies are trying to put AI agents into today.

If you want an agent to work through internal data, financial documents, reports, policies and multimedia artifacts, you need a test where all of that is already mixed together. And where an error means a wrong table that someone may act on.

The benchmark shows several practical things:

🟠 one smart model is not enough

🟣 the design of the agent scaffold changes the result nearly as much as swapping the model

🟠 multimodal integration is still the bottleneck

🟣 evaluation has to account for types, numeric precision, row order and equivalent formats

One more conclusion: reliable analytics needs not just generation but a strict checking system. The authors built a deterministic evaluator that compares tables by substance rather than by the literal look of the headers. That is close to what production will need too.

The takeaway

DataSpace shows a simple fact: AI agents handle real analytics noticeably worse than isolated demo tasks. When the data is scattered across SQLite, documents, tables and video, and the answer has to come back as an exact final table, quality falls off fast.

A best result of 66.34% means a dependable working mode is still some distance away. The main bottlenecks are already visible: integrating several modalities, table joins, cross-lingual matching and materializing the answer exactly.

For anyone building AI agents this is a direct pointer. The next step is improving not only the base models but the agent scaffolds, the tools, the output rules and the control over the form in which the agent hands back its result. In analytics tasks that is what decides whether you end up with a useful assistant or a system that confidently gets one task in three wrong.

🟠 Best result: 66.34%

🟣 Main bottlenecks: multimodality, joins, cross-lingual matching

🟠 Reliability depends on how the agent is built, not on the model alone