PayPal raises AI answer accuracy to 95% without changing the model

What knowledge the assistant needs

PayPal’s internal assistant is designed to answer questions that would normally go to a team of data specialists. Access to records alone does not guarantee a correct answer. For example, an agent might find account records but misidentify which ones count as “new.” It needs business rules to get that right.

PayPal built a knowledge layer designed to work with different models. It includes:

Zhang considers building this knowledge layer a key part of PayPal’s intellectual property.

The knowledge stays, the model changes

Zhang designed the assistant from the start to be independent of any particular model: when a model improves, it can be replaced while the knowledge layer stays in place.

For a separate merchant-facing agent that works with PayPal customers, the team tested four or five models. The larger models had broad capabilities, but initially took about 15 seconds to respond.

Switching to a model with 8 billion parameters significantly reduced latency, but lowered accuracy. Zhang’s team adapted the model for commerce. Accuracy improved, but still fell short of the desired level.

PayPal then began relying less on extensive additional model tuning. It can improve results, but requires ongoing work: it calls for specialized expertise, and the process has to be repeated when new models come out. Zhang said frequent training is expensive. PayPal is not abandoning tuning altogether, though: the team weighs its benefits against the cost of maintaining it as models and workloads change.

How agents use knowledge and tools

PayPal is building a platform that gives agents access to enterprise knowledge while also letting them retain procedures, check proposed solutions against authoritative sources and determine which actions require human approval. Deepika Khera, PayPal’s senior director of AI, described it at the VentureBeat Agentic Infrastructure Stack Summit on October 28.

These mechanisms are especially important in incident investigations. PayPal built a workflow with seven agents to investigate alerts in a system that runs more than 33,000 services. It gathers information from Datadog, ServiceNow and internal logs to separate meaningful signals from noise, assemble context and analyze incidents.

How the work is divided depends on the agents’ tasks, the context they need and the tools available. The number of agents is chosen for each task: another workflow, for security checks, uses up to 18.

Dividing the work does not eliminate the risk of a faulty diagnosis and an incorrect fix. PayPal therefore checks each proposed solution against the original source. The agent’s memory combines certified data sources and standard operating procedures (SOPs); policies and risk assessments provide additional checks. Many decisions still need human review: PayPal is cautious about giving agents autonomy.

Even when a model finds the right information, it may not know how to apply it according to procedure. Khera gave the example of an internal tool for answering questions from HR employees. It struggled because the procedures were stored in a format that did not support accurate answers. The team changed how it stored the instructions, keeping sequential steps together instead of splitting them into separate fragments.

PayPal is also testing when an agent should consult memory. The company combines Model Context Protocol (MCP) integrations with direct API calls. MCP works for some interactions with tools, when an agent needs to choose which one to use. But in a software development workflow, PayPal needed to guarantee that memory would be consulted at a specific stage rather than leave that choice to the agent.

The company is also experimenting with updating memory as agents work. Episodic memory records sessions and pull request (PR) results. At night, the system summarizes the experiences of different users and selects lessons to retain for future interactions. PayPal wants to turn accumulated experience into reusable guidance.

Portable knowledge needs to be tested in practice

Tests may not reflect real-world operating conditions, Zhang noted. A system might answer expected test questions well but fail on the questions users ask most often. If those queries dominate in production, test scores may not match the user experience, even when the model and process have not changed.

The findings of an October VB Pulse survey on agents and the reliability of their evaluation point to the same problem. Among respondents whose organizations run checks before deployment, 42% said that in the past year an agent or large language model feature had passed internal tests and then failed. Of those reporting failures, 49% said the most serious incident affected customers.

Monitoring a system after launch does not always reveal the problem: in the same survey, 39% of respondents did not list automated checks or monitoring signals among the first ways they would detect an incorrect or poor-quality AI response. Instead, they said, people notice the problem.

To better account for the differences between testing and real-world use, PayPal runs broader offline simulations, generates synthetic questions and tests both single-message and multi-message interactions. Zhang also sees value in combining model-based evaluation with human review. After launch, employee feedback helps uncover errors.

According to Zhang, over 10 weeks the internal assistant handled more than 1,000 questions and, by the company’s estimate, saved 1,600 hours. She cautioned that the number of questions alone does not prove a productivity gain.

Errors are harder to detect in consumer systems: users may not explain what went wrong. Repeated queries and users ending a conversation can point to where a problem might be, but do not by themselves prove that an answer was wrong.

Dataist’s take: PayPal’s results show that an assistant’s accuracy depends on more than model choice. It also needs well-organized business knowledge, controls on its actions and tests that resemble real use. But a portable knowledge layer does not remove the need for checks after launch: tests may miss users’ most common questions, and PayPal still leaves many decisions for people to review.