Synthetic Consumers Before Real Ones: Averages Can Hold, Differences Between People Often Do Not

The problem

The sample is the weak point. In a report published in August 2026, Pew Research Center screened an online opt-in survey of 11,114 U.S. adults, fielded in November 2024. A commercial prescreening tool flagged 5,369 cases, nearly half of all completed interviews, and simple trap questions flagged 1,963, or 18% (measured by Pew)³. Bogus respondents distort particular groups. In a December 2023 opt-in poll reported by The Economist, 20% of adults under 30 agreed that the Holocaust is a myth; in Pew's January 2024 probability-based panel of 5,140 adults, 3% of adults under 30 did (measured)⁴.

The other end has thinned too. Pew's telephone response rate was 36% in 1997 and 9% in 2012 (surveyor-reported; the first figure is older than three years)², and it has stayed around 9% since¹. Pew concluded that low response does not by itself make a poll unreliable, and described the resulting bias as limited in scope, but phone polls greatly overstated civic engagement: 46% said they worked with neighbours, against 8% in a government benchmark survey (measured, 2017 report)¹². Weighting corrects most demographic imbalances².

Cost is the stated motive, but it is not measured. One preprint describes cost reductions of 97% as the pitch for synthetic panels⁸. No independent measurement of what a synthetic panel saves was found. Almost all the evidence below comes from the United States.

What already works

Nothing found shows a simulated panel replacing a real one for a real decision. The evidence shows building blocks, and they work best on averages.

Averages can line up. In a peer-reviewed study, ChatGPT personas gave feeling-thermometer scores for 11 sociopolitical groups, and the mean scores corresponded closely to the 2016–2020 American National Election Study averages (measured, abstract)⁵. A preprint tested a method in which the model writes a text answer and software maps it to a 1–5 rating by comparing it with reference statements. Against 57 personal-care product surveys with 9,300 human responses, it reached 90% of human test-retest reliability, meaning the consistency of the same people answering twice; asking the model directly for a number gave unrealistic distributions (abstract, preprint, one company's data, operator-reported)⁶. Another preprint built agents from the interview or survey answers of 1,052 Americans. On held-out General Social Survey items they reached 83% (interview-based) or 82% (survey-based) of participants' own two-week consistency, against 74% for agents given demographics alone (measured, abstract, version dated 28 June 2026)¹⁰.

In plain terms: in the settings tested, a carefully built simulation can come close to how consistently real people repeat themselves, and rich personal information adds more than demographics alone. None of this shows that it works for a new product, in another country, or for what people actually buy.

What does not exist. No study was found that compared simulated respondents with what people went on to buy; the closest tests stated purchase intent⁶. No shared method for validating a simulated consumer population was found in the sources opened.

Where simulated people part ways with real ones

Six studies looked at differences between people rather than at averages.

Aggregates can look acceptable while the variation between people has been compressed⁸. For a business that sells to segments rather than to an average, the spread around the average is the useful part.

The possibility

One way this could work is as a screening stage in front of conventional research. A consumer-goods team with a long list of product concepts could build a population recipe from public survey and census data, and from its own consented customer research where rules allow. Each simulated respondent would be a persona card, and a language model would read a persona and a concept and write a reaction. The simulation would be used to cull weak concepts and sharpen questions, not to report results. Real respondents would then be fielded on the shortlist. A small human sample could serve as calibration: one preprint proposes a correction method that needs 50 to 300 human responses⁸.

The role of AI. Only one step depends on AI: the language model that reads a persona and writes a reaction. Everything else is ordinary: building the population recipe, weighting, questionnaire design, screening for real respondents, calibration statistics, model version control, disclosure and the decision about what to field. If the honest conclusion is that most of the value comes from the calibration and the screening discipline, with the model as a cheap generator, that would be a fair reading.

What would have to be true

1. Fidelity would have to hold for subgroups, not only for averages. One preprint reports subgroup errors of 10–30 percentage points behind acceptable aggregates⁸.
2. The spread of answers would have to be preserved. Studies report less variation than real surveys⁵ and under half the natural spread even after steering⁷.
3. Results would have to be stable. The same prompt gave significantly different results over a three-month period, and minor wording changes altered results (abstract)⁵.
4. The simulation would have to be calibrated against something that really happened, ideally purchases, not only stated intent⁶.
5. The benchmark would have to be the real panel as it is, including its bogus respondents, which Pew shows can be partly screened out³.
6. Use would have to be disclosed. A supplier-authored opinion post warns that clients may not realise that their data is AI-generated¹².
7. Use would have to stay at the early stage. MRS's managing director is quoted as saying that the need for real people will remain¹².

What could stop it

What the evidence shows

• The variance and subgroup failures in the table above recur across independent studies^57891314.
• Results change with prompt wording and over time. Closed models can be updated without notice, so a validated panel can drift (abstract)⁵.
• MRS's chief executive listed among the challenges "normative results which can flatten diverse groups" (May 2024)¹¹.
• MRS's managing director is quoted: "In order to create synthetic data, you still need really, really good human data"¹². The same post says the industry lacks firm regulation¹².
• Simpler fixes exist for real panels. Trap questions cut the share answering yes to at least 10 of 15 yes/no questions from 7% of adults to 1%, and probability-recruited panels are generally immune to this threat (measured)³. Voter-file matching increased error³.

Reasoning

The first challenge is who bears the risk. If a cheap simulation is wrong in a narrow segment, the cost may fall on the brand that launches for that segment, not on the supplier that sold the speed. The second is incentives: suppliers would gain from speed and from reporting the average, which is where simulations look best.

If synthetic respondents were validated only against opt-in panel data, they could inherit that panel's errors, including the bogus answers described above. It is also possible that better screening, weighting and a small probability sample would deliver most of the benefit with no simulation, and the evidence does not rule that out.

Signs to watch

• A published comparison of synthetic-panel predictions with realised sales for products later launched; none was found.
• Replications of the purchase-intent and individual-simulation results by authors with no supplier tie, in other countries and categories⁶¹⁰.
• Peer-reviewed publication of the preprints cited here; the dates are not known.
• New professional-body guidance on disclosing AI-generated respondents; no ESOMAR or AAPOR text was located.
• Suppliers publishing model versions with repeat tests over time, which would address the drift in⁵.