Retail Product Recommendations: Everywhere in Production, but Gains Are Smaller Than Claimed

The use case

Zalando, the European fashion retailer, trained one of its ranking models on the histories of 71 million customers across 25 markets, then tested it in four online A/B tests (live comparisons of two versions). Together the tests raised combined net revenue by 2.2% (operator-reported, A/B test, paper revised 29 October 2024)¹. That is the typical shape of this use case: a model learns from past browsing and buying which products to show each shopper, on home pages, product pages, baskets and emails.

What counts as AI here. The channel counts learned models, such as matrix factorisation or deep neural networks, that rank items from behaviour data. Older tools do similar work and are not credited as AI: hand-set rules, bestseller lists, and simple co-purchase counting ("people who bought A also bought B"), which is statistical and only borderline machine learning. Several of the controlled trials below test "collaborative filters" of this older kind, so their results describe recommendation as a practice, not only modern machine learning. Search relevance, dynamic pricing and display advertising are neighbouring uses; two operator papers below blend them in and are labelled where they do.

Where it is running

The evidence is geographically thin. Controlled trials come mainly from one North American retailer, and the Europe and China results are operator-authored. No operator figures were found for Rakuten, Otto, Mercado Libre, Flipkart or Coupang.

What the evidence shows

Randomised trials. A randomised field experiment at a top North American retailer, covering 82,290 products and 1,138,238 users, found that standard collaborative filters were associated with lower sales diversity than a world without recommendations (controlled trial, abstract, March 2019). Niche items gained sales in absolute terms but lost market share². A later randomised trial of recommending a product category the customer had never bought from found a purchase probability up to 35% higher than no recommendation, 11% higher than recommending the most popular new category, and up to 9% higher when several categories were shown (controlled trial, abstract, working paper revised October 2023; retailer and sample size not stated in the abstract)³. A causal analysis of a field experiment reported that personalised recommendations raised the propensity to buy by 12.4% and basket value by 1.7% (controlled field experiment, abstract, 2022; sample size not verified)⁴. The gap between a large lift in purchase probability and a small lift in basket value matters: recommendations mainly change what people consider, not how much they spend⁴.

Operator tests. Zalando's four A/B tests combined to +2.2% net revenue and a 15% combined engagement uplift, but revenue effects in individual tests ranged from -0.46% to +0.86%, and one search test showed +0.15% revenue that was not statistically significant (operator-reported, A/B test, 2024)¹. Engagement moved much more than revenue. Alibaba reported up to +10.0% click-through and +3.8% revenue per thousand impressions in a test of almost a month in 2017 for display advertising rather than organic recommendations (operator-reported, A/B test, over three years old)⁷.

Compiled company results. A peer-reviewed review of published business results lists a 35% sales lift for purchase-based filtering at a DVD retailer, a 4.4% conversion increase at Booking.com, and a direct revenue gain of only 0.3% at a grocery retailer (as reported in the review, December 2019)⁶. The review warns that click-through gains are often an overestimate of true value and that a gain can come from clicks taken from elsewhere on the site⁶.

What it takes: cost, integration, governance

Software cost. Cloud providers publish list prices. AWS charges $0.15 per 1,000 inference requests, $0.002 per 1,000 interactions for training and $0.05 per GB of data ingested for its current recipes (official price list, page updated 15 October 2026)⁹. Google Cloud lists $0.27 per 1,000 recommendation predictions for the first 20 million (official price list, accessed 17 October 2026, page date not shown)¹⁰. These are list prices for the model service only. Total cost, including data engineering and staff, was not found in public sources.

Data and testing. The models need interaction logs at scale; Zalando's training set held 71 million customers¹. Proving value needs careful testing: the review notes that tests are often short, can miss longer-term effects, and that sample sizes were often small⁶. A study at a Czech travel agency (4,287 users, one month, 2018) found offline accuracy rankings correlated with online results for novice users but reversed for more senior users, so a model that scores well on past data may not win in production⁸.

Rules. In the European Union, the Commission asked Amazon on 5 July 2024 for information on recommender-system transparency, the signals used, and options for users to opt out of profiling, with a reply due 26 July 2024¹¹. It made a similar request to Shein on 6 February 2025, with a reply due 27 February 2025¹³. On 28 May 2026 it fined Temu EUR 200 million because its risk assessment did not properly assess how its design, including recommender systems and affiliate influencer promotion, could amplify illegal products¹². That fine concerns systemic risk assessment, not recommendation accuracy. The UK Competition and Markets Authority warned in April 2022 that combining machine learning with online choice architecture can increase impact and potentially exacerbate harm¹⁴. The text of the Digital Services Act articles could not be read, so the legal detail is taken from the Commission's own press releases.

Where it falls short

Much of the activity is not incremental. An observational study of 2.1 million Amazon.com users over nine months (published 2015, instrumental-variable method, no randomised control group) estimated that at least 75% of the click-through activity linked to recommendations would likely have occurred without them⁵. It covers more than 4,000 products and is over three years old, and no newer equivalent was found. The mechanism is simple: a recommendation often points to something the shopper was already about to find.

Headline shares have no published method. The review notes that Amazon's chief executive said in 2006 that about 35% of sales came from cross-sales, a figure it calls often reported in online media, and that a value of more than USD 1 billion a year is cited for Netflix, with the estimation method "not specified" (operator statements, no method)⁶. These figures should not be used to size the benefit.

Engagement is not revenue. In Zalando's own tests, engagement gains were several times larger than revenue gains, and some revenue changes were negative or not significant¹. The review makes the same point about click-through rates⁶.

Uneven effects. Collaborative filters concentrated sales on popular products, even as individual shoppers saw more variety². The new-category trial found personalisation helped less-recent customers more and that extra choices added no sales once recommendations were personalised³.

Challenge question. What makes the use case look better than it is? Short tests, operator authorship, click metrics, compiled company numbers with different baselines, and the 2006 and Netflix shares all inflate it. Against that, randomised trials do show real purchase lifts. How much depends on data quality, page placement and testing discipline rather than the model could not be isolated from the sources read. No withdrawals, pauses or lawsuits specific to retail product recommendations were found; the enforcement found concerns transparency and risk assessment.

Phase: In wide use — why

The hypothesis of wide use is kept. Confirmed machine-learning systems serve millions of customers a day at Zalando¹ and carried the main traffic at Alibaba in 2017⁷, and AWS and Google Cloud sell managed services⁹¹⁰. The share of all retail recommendations that rely on machine learning rather than simpler rules could not be established, so the phase rests on these confirmed cases. Wide use is not the same as proven value: the evidence of incremental revenue is mixed.

What to watch next

• A Commission decision on Amazon's recommender-system transparency: the 2024 request said the next step could be formal proceedings under Article 66 of the Digital Services Act¹¹.
• Peer-reviewed publication of the new-category trial, which would name the retailer and the sample size missing from the 2023 working-paper abstract³.
• Any independent replication of the Amazon.com finding that at least 75% of recommendation activity is not incremental, which dates from 2015⁵.

Questions to ask before adopting

• What share of recommendation-driven sales would have happened without the recommendation, measured in a randomised holdout?
• Is success measured in revenue and margin, or only in clicks and engagement?
• How long will the test run, with how many shoppers, and will it capture effects beyond a few weeks?
• What happens to sales of niche and own-brand products when popular items are favoured?
• Can shoppers see the main parameters and switch off profiling, as EU regulators now ask?