In February, the astrophysicist and science communicator Paul Sutter stood in front of a group of colleagues and presented a rebuilt version of his algorithm for finding cosmic voids — the near-empty regions between galaxies. The new version ran ten times faster than the old one, handled the quirks of a real dataset far more carefully, and could process surveys a hundred times larger. About ten minutes in, a colleague stopped him: something about the algorithm looked wrong. It was. The code was mishandling the boundaries of the observational surveys. Much of the update had come out of extended sessions with an AI coding assistant, and Sutter wrote up what happened in Nautilus earlier this month.
Sutter is careful about what kind of error this was. Not a typo, not a missing reference, not an arithmetic slip. It was subtle and it was structural: everything computed downstream of it was wrong. By the time the colleague spoke up, Sutter had already walked the whole piece of work through for people who trusted him.
Survey boundaries are a bad place for this to go wrong, and a predictable one. Boundary handling in a galaxy survey is the part of the problem that is specific to the instrument, the sky coverage and the particular dataset — exactly the territory where a language model has the most plausible-looking code available to it and the least ground truth. The code that fails here does not look broken. It looks like code.
Sutter had considered the model's help indispensable. The tool, he says, "sounded like it understood everything," when it was in fact an elaborate next-word predictor.
He is unsentimental about where that confidence comes from. It is not an emergent mystery. People deliberately trained these systems to sound fluent and sure of themselves, the way wolves were bred into dogs that watch a human face for signs of a treat. His question — how do you use a tool that is sometimes wrong and always trying to please you, and how do you trust it — gets a one-word answer: don't.
His analogy for the current moment is alchemy: the thing people did before chemistry existed. AI, on his reading, is at the pre-chemistry stage, and nobody is going to put the instruments down, any more than the alchemists abandoned their sealed crucibles. What he changed after February was procedure. He now traces the model's chain of reasoning step by step, verifies every result, and starts from the assumption that any claim it makes is false.
The context that makes this more than one researcher's bad afternoon is that the literature is already filling with poorly checked, unedited AI-generated material. Researchers are now answerable for whatever a model invents under their name.
I would read this story differently than it is usually told. It is not mainly a story about an AI that lied. It is a story about how close a fundamental error came to leaving the building, and about the single mechanism that stopped it: a human being in the room who was paying attention ten minutes into a talk. That mechanism does not scale, and it is not applied evenly. Code that gets presented to skeptical peers gets caught. Code that goes straight into a pipeline does not. The speedup Sutter was demonstrating — ten times faster, a hundred times bigger surveys — is precisely the kind of claim that buys a result the benefit of the doubt rather than a second look.
What the account does not say is what happened next. How far the wrong version travelled before February, whether anything that depended on it was corrected, and whether anyone besides Sutter changed their own process — none of that appears. His resolution is personal: he will distrust the model. That is a fix for one person's workflow, not for a field.
There is one more detail, and it is uncomfortable rather than damning. The publication that wrote up his story ran Sutter's Nautilus essay through Pangram, an AI-detection tool, which estimated that 56% of the text was probably model-generated. Detection services of this kind are unreliable, and the figure proves nothing on its own.
Sutter's own prescription — check every step, trust nothing — is the correct one and also the expensive one. It gives back most of the time the tool was supposed to save, which is why almost nobody will follow it, and why the next error like his will be found by whoever happens to be sitting in the room.