BSC-Nav's three-layer memory lifts robot navigation to 78.5% success on HM3D

Most AI agents today are reactive: they see a frame and act, see the next frame and act again, and never build a coherent picture of the space around them. Hence the trouble with long routes, with reusing past experience, with flexibility. Biology solved this elegantly: the brain keeps landmarks, route knowledge and survey maps. BSC-Nav carries that principle over to robots and gives them a real cognitive map, so that navigation becomes deliberate rather than moment-to-moment.

What the authors built

BSC-Nav is a memory scaffold modeled on the human brain, with three layers:

• Landmark memory: sparse but reliable cue-to-coordinate associations, each with a description and a confidence score.
• Cognitive map: a voxel representation of space into which features from different viewpoints and different moments in time are carefully accumulated.
• Working memory: a smart mix of the fragments that matter for the current goal, followed by planning.

To see the world, the system uses DINOv2 for visual features and a detector for objects, while an LLM ties the semantics of the query to what is already in memory. The key step is surprise-driven map updating: not everything goes into memory, only what is new and useful. That saves space and makes the system more robust.

How it works inside

For simple category-level goals ("go to the table"), it is enough to check landmark memory and return coordinates immediately. For finer queries ("the round sofa with the light cushion"), the system first expands the text with attribute descriptions, imagines visual prototypes with a diffusion model, and looks for matches in the dense visual features of the cognitive map. The output is a handful of locations scored by confidence and distance. Which one to try first? BSC-Nav ranks candidates by a combined confidence-plus-proximity priority, which saves a great many steps. Underneath sit reliable planners and goal verification.

What the experiments showed

The authors ran more than 8,000 episodes in Habitat on MP3D (the classic dataset of indoor scans of homes and offices) and HM3D (a newer, larger set of scenes for the Habitat simulator, harder and more varied than MP3D), and compared against strong end-to-end and modular baselines. On category goals BSC-Nav reaches 78.5% success on HM3D and 56.5% on MP3D, well clear of the competition. At the instance level it nearly doubles the success metric on Text-Instance and takes 71.4% on Image-Instance, 11.4% above the closest comparable method. Path efficiency is consistently better thanks to sound candidate ranking: goals are often reached on the first try.

Smarter skills

BSC-Nav goes confidently beyond simply reaching a goal. Given long natural-language instructions, the model decomposes the task into a chain of waypoints and follows them. On VLN-CE R2R (where a robot must get from point A to point B in an unseen 3D scene using a text instruction) that comes to 38.5% success at a record efficiency of 53.1%. On active embodied question answering the system scores 54.6 on LLM-Match, and it is especially strong where landmarks have to be tied to spatial context.

Out of the simulator

The team mounted the system on a mobile robot and drove it across two floors. Over 75 episodes with a range of goals, BSC-Nav held at least 3 successful attempts out of 5, and on image-specified tasks four of the five goals were reached 100% of the time. Average speed was 0.76 m/s, and the final distance after stopping was under 2.5 m. On top of navigation the robot carried out simple manipulation: grasping, carrying, placing.

Why it matters

BSC-Nav's main achievement is the shift from reactive policies to layered spatial reasoning. Landmark memory gives fast semantic recall, the cognitive map gives a solid skeleton of the world, and working memory assembles knowledge flexibly around the task at hand. The surprise rule keeps storage compact, while the voxel buffers carefully gather the rare but important views of a scene. The result is an agent that holds in mind not only what and where, but how best to get there right now.

There is groundwork left for later: dynamic and messier spaces, tighter memory budgets, teams of several agents, new sensors. But BSC-Nav already shows how biological principles can be turned into a scalable path toward general spatial intelligence for embodied systems.

💾 Code

🎥 Video