Agents need less memory when the environment keeps the traces
In AI we are used to thinking of memory as something that sits inside the agent: in an RNN's hidden state, in the network's weights, in a replay buffer, in the KV cache. But what if part of that memory can literally be moved outside — into the environment itself? Not in a metaphorical sense but a formal one: so that the agent genuinely needs less internal memory when the world leaves useful traces of the past.
When memory lives in the world, not the model
The paper Artifacts as Memory Beyond the Agent Boundary is about exactly this. The authors take an old idea from cognitive science — that the mind leans not only on the brain but on external supports such as notes, bookmarks and traces — and restate it as a reinforcement learning problem. The result is an intriguing claim: the environment can do the work of memory, and certain observations let an agent "recall" the past without storing all of it internally.
It sounds philosophical, but the work is surprisingly practical. There is a formalism, a theorem, and careful experiments with Q-learning and DQN. More to the point, there is a conclusion that matters for ML: sometimes you improve performance not by adding parameters but by organizing the environment the agent learns in.
What "artifacts" as memory actually means
Picture an agent walking through a maze and leaving a trail behind it, like breadcrumbs. Later, seeing that trail, it gets information about where it has already been. Part of the interaction history is now encoded not in the agent's internal state but in the outside world.
The authors call observations like this artifacts. In their formal definition, an artifact is a current observation that establishes with full certainty that some other observation occurred in the past.
To explain it they use an almost domestic example involving a book: you don't memorize the page number, you fold the corner. The folded corner is the artifact — a physical trace in the environment that stands in for remembering internally.
The key idea of the paper is that this is more than a pretty analogy. The authors show mathematically that if a history contains artifacts of this kind, predicting the future can require less information about the past.
Why this matters for reinforcement learning
First, the authors argue that part of a task's difficulty can be taken off the agent and put on the environment. If the environment supplies useful traces of the past, the agent does not have to hold as much memory internally.
Second, this matters for embodied AI and robotics. The real world is full of external supports: labels, maps, tracks, the state of objects, interfaces, even where things sit on a desk. An agent that can use such signals as memory may be cheaper and more reliable than one that always builds everything out of internal memory.
Third, the work offers a new way to look at "memory" in reinforcement learning. Usually the term means recurrent state, or simply parameter count. Here memory is treated functionally: how much internal capacity it takes to reach a given level of behavior. If artifacts lower that threshold, then part of the memory really has been moved into the outside world.
The central theoretical idea
The theoretical core is the Artifact Reduction Theorem. Translated into plain language, it says roughly this: if the interaction history contains an artifact observation that already encodes information about some past event, that part of the past can be dropped from the representation of the history without losing relevant information.
The authors state it through mutual information: there exists a shortened history, one observation shorter, that carries as much information about the next observation as the full history does. An artifact, in other words, works as a compact external record of the past.
They then define externalized memory. An agent counts as externalizing memory if, in a world with artifacts, it reaches the same or better performance with less internal capacity than a comparable agent in the same environment with the artifacts stripped out.
That is an important detail: memory here is measured not by introspecting hidden states but operationally, through the required capacity — that is, the number of trainable parameters.
How the researchers tested it
The experimental section is put together rather elegantly. There is a navigation task on a 13×13 grid: the agent has to find a goal, with a reward of +1 for reaching it and zero on all other steps. The observation is not coordinates but a 24×24 image showing the local patch of space around the agent. So the task is not trivial — it is partly a perception problem.
Two groups of agents are compared:
The baseline is No Path, where there are no useful traces at all. Against it they set variants in which some spatial structure is visible in the environment: the shortest path to the goal, a random path, a suboptimal path, a misleading trail, or a set of geometric landmarks.
Importantly, the agent is never given a separate objective of "use the path as memory". It simply learns by reinforcement.
When a visible path replaces internal memory
The strongest result is the shortest-path experiment. When a trail to the goal is visible in the environment, agents need less internal capacity to learn a good policy.
For both linear Q-learning and DQN, a visible path almost always improves final reward. And in a number of cases a small agent with the path does better than a higher-capacity agent without it. That is the evidence that external memory is at work: the environment compensates for a shortage of internal resources.
The learning dynamics are more interesting still.
Without a path, low-capacity linear agents effectively fail: reaching the goal consistently takes substantially more capacity. With the path, it no longer does. For DQN the effect is positive almost everywhere as well.
Why does this happen? The authors offer a simple explanation: if the shortest path is present in the observation, the agent does not have to reconstruct a long history of movement or build a complicated internal representation of space. It is enough to learn to react to local visual features of the path. The environment itself indicates where the agent stands relative to the goal.
Not just perfect hints: random paths and landmarks work too
To rule out the possibility that the agent is simply walking along the correct route, the authors run a second set of experiments with other types of artifacts: a random path, a suboptimal path, a misleading trail, and a set of geometric landmarks.
If the effect were explained only by drawing the agent a near-finished solution, then bad or irrelevant trajectories should not help much. But they do help — though not equally.
Linear agents show signs of external memory for all four artifact types. With DQN the effect holds consistently for the suboptimal path, the random path and the landmarks, but not for the misleading trail. That is logical: not every external structure is equally useful. Still, the fact that even markers carrying no explicit "behavioral" hint help is quite telling.
This means the agent is probably using not only a ready-made "go here" instruction but any stable element of the environment as an anchor for localization and for recovering context. In other words, external memory need not be a literal trace of behavior — it can simply be structure that reduces uncertainty about the past.
The most interesting experiment: the agent leaves its own traces
The third experiment takes a step toward a more realistic picture. Here the path is not drawn in advance. It appears as the agent moves: visited cells leave a noisy trail that then gradually fades.
This is close to stigmergy in miniature: the agent acts, changes the environment, and then perceives the result of its own past actions as a signal for future behavior.
The researchers show that even without a dedicated mechanism for writing to external memory, an agent can create it unintentionally and then use it. Memory emerges not as a hand-designed module but as an effect of the interaction between policy and environment.
Takeaways
The researchers do not merely say that "the environment matters" — they show that the environment can functionally play the role of memory, reducing how much internal memory an agent needs to behave well.
If the surroundings hold stable traces of the past — routes, landmarks, markings, objects that change state — an agent can use them as external memory, and it can do so without any dedicated write mechanism. That changes how intelligent systems get designed. Sometimes the route to a smarter agent runs not through another billion parameters but through an environment that helps it remember.
For reinforcement learning this matters especially. We talk a great deal about scaling models, but we may be underrating scaling the environment — how it structures the task, supports memory and nudges the agent toward effective behavior.
AI paper breakdowns
Every day we read the new AI papers and retell what matters in plain language — no hype, no filler. If you want to see where AI agents are heading before everyone else, subscribe.
New breakdowns every day
On Telegram