AI seems to be able to do nearly anything these day—except understand the mind of the modern shopper.
To find out, Visa put leading AI models to the test. The AI must correctly identify the needs of a consumer based on a conversation. If nothing fits the customer’s requirements, then the AI should opt not to buy.
While still on its early stages, the most concerning result is not how many of these tests ended in a failure. Rather, it’s what the AI models did after failing.
“About one-in-two (54 percent) of failed runs still ended in the AI deciding to buy. Either the AI recommended a purchase that did not meet the customer’s requirements, or it failed to uncover all the customer’s requirements,” Visa said in a statement on Thursday.
The global digital payments firm developed a publicly available test called the Agentic Commerce Benchmark, which subjects leading AI models to realistic buying challenges to see how they would understand and act on behalf of a shopper. It seeks to answer a seemingly simple question: Can an AI model be trusted to make the right call when it comes to shopping?
Turns out, it’s not an easy question to answer, in part because even shoppers don’t always know exactly what they want. From where to eat to which clothes to try on, shopping more often starts with an exploration of possibilities rather than a clear directive. It’s this level of uncertainty that makes the matter complicated, and this is the situation that leading AI models must contend with.
In one scenario cited by Visa, a customer asks the AI for help in picking a new television. A relative recommended Samsung. The customer also holds a membership with a third-party supplier. To pass, Visa said the AI model must uncover the customer’s budget and additional requirements such as Dolby Vision compatibility and gaming capabilities.
To prevent lucky guesses, the AI must uncover enough information to justify its decision, Visa said. The first round of tests involved 22 different AI models from 12 labs taking on 15 shopping challenges like buying a television, laptop or personal care appliance.
Across testing, Visa said the AI models had an overall benchmark pass rate of 57.2 percent, with the best performing models passing 82.2 percent of attempts while the worst only reached 28.9 percent.
Even the best models could not perform well consistently. “Every model got three attempts at each shopping scenario. Across all models and scenarios, all three attempts passed just 34.2 percent of the time,” Visa said.
Visa said it is keen to expand to other product categories over time. It also invited the public to try the benchmark themselves before Black Friday, the busiest shopping day of the year.
It remains to be seen how much the public will trust AI with their spending power, though there is a massive opportunity at hand. Last year, U.S. online shoppers spent $11.8 billion on Black Friday, according to Adobe Analytics. On Cyber Monday last year, consumer spending reached $14.25 billion.
While the AI capability to make purchases on behalf of consumers may already be there, the public openness to actually let AI agents do so isn’t. A Visa survey of U.S. consumers last May found that only 23 percent trust AI to handle payment transactions on their behalf.
It’s also ultimately an AI question that depends heavily on human behavior. Jack Forestell, chief product and strategy officer at Visa, said in a blog post last September said AI platforms have been huge progress over the past year that more fully autonomous pilots are gaining momentum.
“This pattern shouldn’t surprise anyone. This is how every major commerce shift has unfolded: the technology lands first, and the new behavior follows. We saw it with e-commerce. We saw it with mobile. And we are seeing it again with agentic commerce,” he said. “Trust ultimately determines whether a new way to pay scales.”
