Instacart's substitution model does two jobs with one number. When a shopper in the aisle cannot find the item a customer ordered, the model ranks the available replacements by the probability the customer will be satisfied — and that same probability becomes the system's confidence in its own pick, the value that decides whether it acts. Ahsaas Bajaj, who leads machine learning at Instacart, calls substitution harder than most machine learning projects because of the size of the decision space: potentially thousands of items for every prediction. His system uses explicitly defined confidence levels that shape decisions at runtime.
Picking the replacement is only half the work. The other half is the system knowing how sure it is, and that is the half the industry has spent less attention on.
The reason it matters is structural. AI is put to work on problems that cannot be solved perfectly — that is close to a working definition of the field: a system attempting a problem where no error-free answer exists even in principle. Generative AI raised the ambition of such projects, and models are now discussed as possible stand-ins for a support agent, an analyst, a teacher, a virtual assistant. Fully autonomous systems at that level look doubtful in the near term, but newer approaches allow large language models to be aimed at more modest targets. Predictive AI has the same problem and is decades older; it went by "predictive analytics" before the relabeling. The methods keep improving, and there is still no universal way to predict without error who will click, buy, lie or die.
If the task cannot be solved perfectly, something has to stop an unreliable prediction from becoming an action on its own.
At Instacart, that something is the score itself. Say a customer ordered a particular brand of honey cereal. The model evaluates candidate replacements on product attributes — brand, department, price, and dietary markers such as organic, vegan, vegetarian or fat-free. The candidate set can include honey cereal from other brands and adjacent products such as granola clusters. For each one the system computes the probability the customer will be happy with the swap. That figure ranks the options, selects the best, and sets the confidence level in the chosen item.
The threshold is where the business lives. Filter too aggressively and the service misses replacements the customer would have accepted. Filter too loosely and the customer sees recommendations with low odds of working out, and stops trusting the service. Tying prediction quality to a legible business decision is one of the places where this technology does visible work.
Bajaj took a master's in computer science at UMass Amherst and worked at Walmart, Goldman Sachs and Samsung before Instacart. The systems he has built support hundreds of millions of operational decisions in real time, which is the scale at which a badly chosen threshold stops being a metric and starts being a customer experience.
The mechanism is not new, and the canonical example is fifteen years old. IBM's Watson, which beat the reigning Jeopardy! champions in 2011, did not merely select an answer — it scored its confidence in that answer, which the game required: buzz in and get it wrong and you lose points. Watson ran a machine learning model that estimated the probability each candidate answer was correct, checked a large number of candidates repeatedly, and chose the one it rated highest. The same predictive score that ranked the candidates served as the confidence in the winner. On one clue — mitosis divides the nucleus during cell division, and cytokinesis divides this fluid surrounding it — Watson answered "cytoplasm" at 92% confidence, the number that both put it first and told the machine it was safe to buzz.
The same pattern now goes by hybrid AI, and it works as a reliability fuse across systems that are not predictive themselves. A predictive model decides which cases go to a human, which makes automation usable in places where, unchecked, it would not be. Hospitals and health networks use AI to assemble insurance claims automatically, then run a separate predictive model that scores each claim's risk of denial; staff review the high-risk ones before the claim goes to the insurer.
My read is that the interesting part of Bajaj's job is not the model at all — it is that he does not set the threshold. He analyzes how different decisions based on the confidence scores move customer satisfaction and puts deployment options in front of the company; several decision-makers at Instacart then choose which substitution to offer. That is the correct arrangement and it is rare. A confidence threshold is a product decision wearing the costume of a hyperparameter: it determines how often the service is visibly wrong, how often it is silently unhelpful, and who absorbs the difference.
What none of these designs announce is that the fuse needs somewhere to send the current. Instacart's works because a human is already standing in the aisle holding the wrong box of cereal; the hospital's works because reviewers exist to read the high-risk claims. Strip out the cheap human fallback and the confidence score degrades into a number the system prints and ignores. That is the quiet precondition under every hybrid AI deployment being sold right now as a way to run with fewer people.
Bajaj's framing puts a ceiling on what a model gets to do alone: act where a second estimate says the odds are good, defer where it does not. That is a smaller claim than replacing the support agent — and it is the one that survives contact with a real grocery order.