An AI agent increased sales by 16.3%.
An AI tool for advertising titles produced no detectable sales increase.
Same company. Different workflows.
That is the uncomfortable part of enterprise AI: a system can do what you asked and still fail to improve the outcome you care about.
A working paper published in October 2025, Generative AI and Firm Productivity: Field Experiments in Online Retail, examined seven randomized experiments at a large international e-commerce platform.
The experiments involved millions of customers and products. Sales effects ranged from no detectable improvement to a 16.3% increase.
The obvious headline:
AI can increase sales.
The more useful lesson:
Choosing where to use AI may matter more than deciding to use it.
Here is what the experiments reveal—and how I would use them to decide what to build.
Seven experiments. No universal AI dividend.
The researchers tested generative AI inside real business workflows. The experiments ran between September 2023 and June 2024. The paper was published in October 2025.
The company was not identified.
Here are four of the applications:
| Workflow | What changed | Reported sales effect |
|---|---|---|
| Pre-purchase support | An AI agent answered customer questions | +16.3% |
| Search-query refinement | AI improved interpretation and translation of queries | Approximately +2–3% |
| Product descriptions | AI added richer, structured information | Approximately +2–3% |
| Google advertising titles | AI optimized product titles | No detectable increase |
These are separate comparisons against each workflow’s existing approach.
They are not additive gains. They are not a company-wide revenue increase. And they are not a guarantee for another business.
The technology category was similar.
The commercial results were not.
To understand the difference, start with the baseline behind that eye-catching 16.3%.
The most important detail is what the AI replaced
The customer-service experiment involved 44,614 customers over two months.
One group could interact with a generative AI agent before purchasing.
The control group received a scripted message indicating that customer service was unavailable.
The AI group generated 16.3% more sales relative to that control.
Now compare these two headlines:
“AI beats customer service by 16.3%.”
“Providing AI assistance beats leaving customers without effective assistance.”
Only the second accurately describes the comparison.
That distinction is not a technicality. It changes the business case.
The experiment did not establish that an agent outperformed an excellent human support team.
It showed that answering questions could outperform an experience where those questions went unanswered.
The opportunity was a service gap.
My takeaway:
Before trying to replace the best person in the company, investigate the useful work nobody currently has the capacity to do.
Search: the products existed. The connection was weak.
The search experiment involved 1,849,382 customers.
The existing process used basic translation. The AI intervention added semantic interpretation to better convey what customers meant.
The study reports a sales improvement of roughly 2–3%.
Think about the engineering problem.
The company already had:
- A product catalog.
- Customer queries.
- A search system.
The intervention targeted the connection between them:
What the customer says
↓
What the customer means
↓
What the search system retrieves
This diagram is my interpretation, not the study’s published architecture.
But it captures an important design choice:
You do not always need to replace an entire experience. Sometimes the useful intervention sits between two existing components.
“Build a shopping chatbot” and “improve query interpretation” are not the same project.
One starts with an interface.
The other starts with a failure.
Product descriptions: sometimes the opportunity is missing information
The product-description experiment involved 4,772,937 customers.
AI supplemented existing product information with more comprehensive, structured descriptions.
The baseline explains why this was worth testing: nearly half of the platform’s self-operated products had limited descriptions or no descriptions.
The study reports a sales improvement of approximately 2–3% for this application.
Again, the starting point was specific.
Not:
“Our organization needs to become AI-ready.”
But:
“Customers are trying to evaluate products without enough information.”
Those statements lead to very different projects.
The first can become an open-ended transformation program.
The second gives you a problem, a user, and a place to measure an effect.
For my own implementation, I would require generated specifications and product claims to be grounded in approved sources.
Filling an information gap should not become an excuse to manufacture information.
That is a design recommendation—not a safety outcome established by this experiment.
The result every AI presentation should include
AI-optimized advertising titles produced no detectable sales increase.
Keep that result in the story.
It is tempting to discuss only the successful experiments. But the non-result is what makes the comparison useful.
A title generator is easy to understand.
Easy to demonstrate.
Easy to admire.
None of those properties establishes incremental commercial value.
“No detectable increase” does not prove that the true effect was exactly zero. It means the experiment did not demonstrate a sales improvement under the tested conditions.
The practical question is:
If the output changed but the outcome did not, what are we paying to improve?
A better-looking artifact is not automatically a better business result.
The variable I would put at the center: the baseline
These experiments made me think less about AI capability in isolation and more about headroom.
By headroom, I mean the improvement available above the current experience.
An unanswered question may offer substantial room for improvement.
A product page with missing information may offer room for improvement.
A workflow that already performs well may offer much less.
This is my synthesis, not a formal metric from the paper.
It leads to a different project-selection question:
Old question:
Can AI do this task?
Better question:
Can AI improve this outcome enough
to justify the cost and risk?
The first question selects demonstrations.
The second selects experiments.
Before you build an agent, write this sentence
Here is the project brief I would use:
Today, [specific user] cannot reliably [specific action] because [specific gap]. We will test whether [bounded intervention] improves [measurable outcome] compared with [current baseline], without exceeding [cost and risk limits].
For a hypothetical retailer:
Customers cannot get compatibility answers outside support hours. We will test a product-information assistant against the current unavailable-support experience, measuring purchase conversion while checking unsupported claims and operating costs.
Notice what is missing:
- A model name.
- A vector database.
- An agent framework.
- A multi-agent architecture.
Those decisions come after the problem is clear.
A stack is an implementation choice.
It is not a business case.
Your first release should answer a question
For a hypothetical pre-purchase assistant, I would define the experiment like this:
SCOPE
One product category
One customer journey
Approved information sources
COMPARISON
Current experience vs. AI-assisted experience
PRIMARY OUTCOME
Purchase conversion
GUARDRAILS
Unsupported factual claims
Customer complaints
Escalation failures
Response latency
ECONOMICS
Incremental contribution
minus total operating costs
DECISION
Scale, revise, or stop
This is a proposed template, not a reconstruction of the retailer’s experiment.
Its purpose is to make the first release answer:
“Is this worth doing?”
Not merely:
“Can we make it run?”
Those are different milestones.
Only one justifies scaling.
What this study does not give us
It does not give every company a guaranteed 16.3% sales lift.
It does not establish that agents should replace human support.
It does not turn sales growth into a complete return-on-investment calculation.
And a paper reporting real operational data is not the same thing as a publicly downloadable raw dataset.
The paper is a working paper about one undisclosed platform. Its experiments provide useful evidence, but the results belong to those settings and comparisons.
The authors also report that sales gains were driven mainly by higher conversion rather than larger average order values.
That is an important distinction.
The interventions mainly helped more customers complete purchases—not necessarily make each buyer spend more.
“AI increased revenue” is a headline.
“AI helped remove a barrier to purchase” is a mechanism you can investigate.
Stop shopping for agents. Start looking for gaps.
The most useful thing about this study is not the biggest percentage.
It is the variation.
Seven applications inside one company did not produce seven versions of the same success story.
So before the architecture diagram, before the framework debate, before the agent demo, ask:
- What is missing from the current experience?
- Who pays the price for that gap?
- What information would help?
- What outcome should change?
- What evidence would make us stop?
Your next valuable AI project might not begin with a more powerful model.
It might begin with an unanswered customer question.
What would you test first in your organization: improving a task that already works—or providing a useful service nobody currently offers?
created by : https://almodarresieh.github.io/
References
- Fang, L., Yuan, Z., Zhang, K., Donati, D., and Sarvary, M. (2025). Generative AI and Firm Productivity: Field Experiments in Online Retail. arXiv working paper, version 1, October 14, 2025. Abstract: https://arxiv.org/abs/2510.12049v1 Full text: https://arxiv.org/html/2510.12049v1
All empirical figures in this article refer to that version of the paper. The engineering templates, interpretations, and implementation recommendations are the author's synthesis, not additional findings from the study.
Top comments (0)