A DAG-editing agent matches script baselines at 42.8% lower cost

When AI builds a pipeline, not a throwaway script

LLMs already write decent code from a natural-language description. In a production setting that is often not enough. A user asks: "build me a pipeline for data cleaning, filtering, question generation and quality checks." The coding agent answers with a Python script. The script runs. Sometimes it even works. Then the familiar pain starts: it is awkward to edit inside the platform UI, it does not live as a persistent object in the system, and it is hard to validate, reuse and hand off to the rest of the team.

The authors call this the gap between "natural-language text" and "a real platform pipeline." They give it a name: the NL2Pipeline gap. And instead of asking the model to write free-form code straight away, they propose making it assemble the pipeline piece by piece, as a structure of operators, edges and checks.

The idea sounds mundane. But it may be exactly what separates a good demo from a tool you would actually wire into a working data platform.

What goes wrong with scripts

Short version: free-form code generation is fine for one-off jobs. It is bad wherever an artifact needs a lifecycle.

In real systems a pipeline is not a text file. It is an object that:

A generic coding agent guarantees none of that. It can reference an operator that does not exist, guess the data format between steps wrong, or invent behavior the platform simply does not have.

So the task here is not to make the model "even smarter." It has to be grounded in a specific environment: shown which operators are available right now, what state the pipeline is already in, and which changes are legal.

What DataFlow-Harness is

DataFlow-Harness is a platform where the AI agent does not write a free-form script but incrementally assembles a platform DAG — an acyclic graph of data-processing steps.

The key point: the agent does not produce everything in one large chunk. It makes typed edits. Adds an operator. Changes a parameter. Connects two nodes. Deletes a redundant step. After every action the system checks that the graph is still valid.

The DataFlow-Harness architecture: the agent, the shared pipeline model, skills, the tool layer and edit validation.

The system has four main parts.

Instead of "here is a prompt, write the code," you get "here is a live platform, an operator registry, the current state and a set of legal actions."

Why skills matter here, not just tools

You might think: if the agent already sees the list of operators, what does it need skills for? Isn't access to the platform enough?

It turns out it is not.

The tool layer shows the agent what is available. It does not explain how this is normally assembled. The model can see a text-extraction operator, an annotation operator, a filter and a quality scorer, and still miss the order of operations, the hidden dependencies and the typical pipeline shapes.

That is what DataFlow-Skills are for. A body of procedural knowledge:

Put differently: tools give facts about the platform, skills give working practice.

What this looks like for the user

One of the most practical elements of the paper is the pairing of the conversation with a visual editor. The user describes the task in words, the agent proposes changes, and the result appears immediately as a graph. From there you can adjust nodes, edges and parameters by hand. On its next turn the agent already sees the updated state.

Two synced working modes: the conversation with the agent and the visual graph editor.

That removes the usual problem with AI tools: "the model generated something, and now either take it as is or rewrite it all yourself." Here the pipeline stays a live object. You can edit it by hand, and the agent carries on from the latest version.

For production work, that is a baseline requirement.

How the system was tested

The authors built a benchmark of 12 data-engineering tasks. It covers a range of scenarios: question-answer generation, long-document processing, junk filtering, schema normalization, multi-field scoring and other routine jobs.

Four setups were compared:

All of them ran on the same base model, so the comparison is about the system around it rather than the difference between models.

The metrics are practical too:

The main results

The numbers are interesting for their balance, not for one peak row in the table.

The full DataFlow-Harness reached 93.3% successful runs. That is nearly the level of the best script-based baseline with access to platform context, which hit 94.2%. The gap is very small. But DataFlow-Harness ran noticeably cheaper and faster.

Against plain Claude Code with no platform context:

Against the stronger baseline, where the agent was given the platform codebase:

The telling part: the tools-only variant, without skills, dropped to 83.3%. Wiring an agent into the platform is not enough by itself. It also needs procedural support.

This is where the NL2Pipeline gap shows itself: writing an executable script is easier for the model than assembling a valid platform DAG. Give it the right constraints and knowledge of the process, and the gap nearly closes.

Where skills actually help

The authors broke the tasks down by type. It is one of the most useful parts of the work.

Skills helped most where success depends on an implicit order of operations. Question generation, for instance, and chained text processing. In those cases it is not enough for the agent to pick "reasonable" operators. It has to understand the assembly procedure.

On simple tasks — rename a field, flatten a nested structure, filter by length — there was almost no difference. Both variants handled them perfectly.

And where the bottleneck is no longer graph construction but the quality of the model's own output or numeric constraints on the result, skills worked no miracles either. That is an honest finding: a good scaffold does not fix everything.

It comes out like this:

A harder case: pulling questions and answers out of textbooks

The authors also tested the system on a more lifelike task: extracting question-answer pairs from teaching materials. This is a rough scenario. The documents are long, the structure is broken, and figures, tables, worked solutions, captions and chunks with different internal logic sit right next to each other.

Here DataFlow-Harness came out ahead on two metrics at once:

The baselines came in noticeably lower. Recall matters especially: the system was not simply filtering its output more cautiously, it genuinely extracted more correct pairs.

That shows clearly where the platform approach wins. On a hard task, "being able to write code" no longer covers it. You have to find and correctly connect specialized components: PDF parsing, page-structure reconstruction, OCR, figure extraction, multimodal analysis, matching a question to an answer far away in the document.

This is exactly where reusing an existing operator ecosystem matters more than trying to generate everything from scratch.

The most interesting test: does this change data quality downstream

The authors went one step further and asked: fine, the pipeline can be built. But does the result at the next stage get better? Say you use that pipeline to generate training data and then fine-tune a model on it.

They ran two such scenarios.

The first is a math pipeline for cleaning and synthesizing problems. The second is generating a general instruction set to fine-tune a model from scratch, including critique, rewriting and automatic quality scoring.

The logic of the test is simple: if the pipeline is assembled better, the data coming out of it should be more useful. Which means the model should score higher after fine-tuning.

That broadly held. In the math scenario, data produced through DataFlow-Harness gave the best average result at an equal training budget. In the general scenario the gain is clearest on coding tasks: a model trained on data from that pipeline did better on HumanEval and MBPP.

This does not mean the platform automatically produces better datasets always and everywhere. The authors are cautious themselves: two cases so far, few runs. But the direction matters. The question is no longer only "can the agent assemble the graph," but "does that graph lead to more useful data."

Why this matters

There is a practical idea in this work that carries well beyond data engineering.

If you want to embed an LLM in a working platform, free-form text or code generation is the start of the job, not the end. You need a layer that:

This applies past data pipelines. The same logic fits internal business processes, visual editors, automation systems, and possibly part of the tooling in robotics or multi-user environments where a shared state object matters.

The core idea here is not "a smarter agent." It is that the agent needs a properly built environment.

The takeaway

The gap between a text instruction and a real platform pipeline is an engineering problem in its own right. A more powerful LLM alone will not close it.

DataFlow-Harness points to a more practical route: the agent should work with a live representation of the pipeline rather than free-form code; with a registry of available operators rather than abstract knowledge; with a chain of validated edits rather than one answer.

The result: roughly the same success rate as the script-based baselines, but noticeably lower cost and latency, and what is left at the end is an editable DAG inside the platform rather than a throwaway script.

For teams that want to use AI as part of a real data-development environment rather than as a code generator for prototypes, that may matter more than another few percentage points on an abstract benchmark.