The Simulated Town as Policy Test Bed: Useful for Rehearsal, Not for Verdicts

The problem

By December 2018, England's Help to Buy equity loan scheme had made around 211,000 loans worth £11.7 billion¹. Only afterwards did the government's own commissioned evaluation, based on surveys of buyers, find that around three-fifths of them could have bought a property without the support, and 31% said they could have bought the property they wanted (self-reported, surveyed between June 2015 and March 2017)¹. The National Audit Office also noted that the department had not undertaken a detailed assessment of the scheme's impact on the wider housing market¹. These figures date from 2019 and are more than three years old, but they show a pattern that is common in policy: how people respond to a new rule is learned after the rule is already in force, from evaluations that arrive years later.

The tools governments use to test policies before introduction mostly do not model that response. EUROMOD, the European Union's tax-benefit model, run by the European Commission's Joint Research Centre and covering the 27 member states, is described by its operator as "static and non-behavioural": it does not take into account behavioural responses of individuals, and its simulations usually assume full benefit take-up and full tax compliance². Such models answer "who gains and who loses if nobody changes what they do". They are not built to answer "what would people do instead".

The idea. This paper explores a possibility: a policy is first tried on a simulated town, a population of software agents that each stand in for a real kind of household, that react to the rule, adapt and influence one another, before it is tried on real people.

What already works

Static microsimulation. Tax-benefit microsimulation is mature. EUROMOD calculates personal income tax, social insurance contributions and most non-contributory benefits for a representative sample of households, using EU survey data, and is used to assess the budgetary and distributional consequences of reforms². This is ordinary software: a precise encoding of the rules, applied to real household records. It is the part of the simulated town that already exists and would not need AI.

Adding behaviour has always been hard. Microsimulation researchers have discussed for decades how to add behavioural responses. A discussion paper written for the EUROMOD preparatory project, republished in the International Journal of Microsimulation in 2022, concluded that for labour supply there were problems with the timing of responses in a static approach and a lack of comparable studies or data across countries³. The argument predates current AI by many years, but the difficulty it names has not gone away.

Agent-based models without AI. Agent-based models simulate many individual decision-makers following explicit rules. In 2016 Bank of England staff and Oxford researchers published an agent-based model of the UK housing market, calibrated on a large set of micro data including loan-level mortgage sales data, Land Registry transactions and rental listings⁴. They used it to test a loan-to-income portfolio limit and found that the policy attenuated the house price cycle within the model⁴. The authors state that the model does not try to represent the market at a specific point in time and is not intended to produce forecasts⁴. The agents here follow hand-written rules, not learned behaviour.

Agents that learn. A 2021 research paper from a company lab (vendor-affiliated research) trained simulated economic agents and a simulated tax-setting planner with reinforcement learning, a technique in which software improves by trial and error against a reward, so that each adapted to the other⁶. In simple economies it recovered the optimal tax policy of economic theory; in more complex ones it reported improvements over baselines despite agents developing tax-gaming strategies⁶. The economies were artificial; no real population was represented.

Agents grounded in real people. The newest building block uses large language models, AI systems trained on vast amounts of text that can generate human-like answers. A team of US researchers built agents for 1,052 Americans from two-hour interviews, survey answers or both⁵. On held-out General Social Survey questions, interview-based, survey-based and combined agents reached 83%, 82% and 86% of the accuracy with which participants reproduced their own answers two weeks later, compared with 74% for agents given only demographics (measured without a control group; preprint, revised June 2026)⁵. The authors report that the richer agents also reduced accuracy gaps across racial and ideological groups relative to demographics-only agents⁵.

In plain terms: an agent built from a long interview with a real person answers attitude questions roughly as consistently as that person does, and better than an agent built from a census-style profile. That is evidence about survey answers, not about how people behave when a real rule changes their money, rent or time.

What does not exist. No study was found in which language-model agents, coupled to a full tax-benefit rules engine, were used to test a real policy before introduction and then checked against what happened. Nor was any accepted method found for validating such a simulation against real outcomes. These pieces would have to be invented.

The possibility

One way this could work: a finance or social policy ministry would keep a simulated town of several thousand agents. Each agent would be built from consented interview and survey data, weighted to match the national population, and would sit on top of a static rules engine like EUROMOD that computes exactly what each household would pay and receive under a proposed rule. Analysts would introduce the rule, and the agents would decide what to do: claim or not claim, work more hours or fewer, move, buy, complain, tell their neighbours. Over simulated months, the analysts would watch for effects nobody intended: support going mostly to people who did not need it, a cliff edge where earning more leaves a family worse off, a group that drops out of a service.

What depends on AI, and what does not. The arithmetic of taxes and benefits is ordinary, rules-based software and should stay that way: it must be exact and auditable. The survey data, consent arrangements, weighting and computing infrastructure are data plumbing and money. The AI part is narrow but central: language-model agents would generate the behavioural response, the step that static models leave out and that hand-written agent rules capture only for behaviours someone thought to write down. Reinforcement learning could add agents that probe a rule for loopholes, as the tax-gaming strategies in the planner experiments suggest⁶.

What it would add over today's tools. If it worked, the main gain would be breadth rather than precision: a way to surface behaviours that no analyst thought to model, such as a cheaper substitute or a new loophole, early enough to redesign the rule. Its honest use would be rehearsal, a structured way to ask "what might go wrong?", not a verdict on whether a policy works.

Feasible in parts. The rules engine exists. Agents that echo real people's survey answers exist in research. Coupling them at town scale, and proving that the result predicts real behaviour, does not exist yet.

What would have to be true

1. Agents must behave, not just answer. The strongest evidence concerns how agents answer attitude and personality questions⁵. A policy test bed would need evidence that agents reproduce real decisions under real incentives, such as take-up, hours worked or moving home, which has not been shown.
2. The simulation must be checked against the past. Before forecasting a new rule, a simulated town would need to reproduce the known effects of rules already evaluated. Help to Buy, with its published evaluation of who would have bought anyway, is the kind of case that could serve as a test¹. If a simulated town cannot reproduce the effects of a policy that has already been evaluated, it has not earned the right to predict the effects of one that has not.
3. The population must be represented fairly. Agents built from demographics alone performed worse and less evenly across groups than agents built from interviews⁵, which implies collecting rich, consented data from a representative sample, including groups that are hard to reach.
4. Results must be stable. Conclusions should not change with the wording or language of the instructions given to the agents, a condition current tools do not meet⁸.
5. Someone must be accountable for the output. A simulation used in policy would need published documentation of how it was built, tested and used, along the lines of the development and deployment reports proposed by researchers⁹.
6. It must beat simpler options. It would need to tell decision-makers something that existing behavioural assumptions, pilots, consultations and randomised trials do not, at a cost that justifies it.

What could stop it

What the evidence shows

• Simulated people flatten real people. A study across 4 language models, with 3,200 human participants from 16 demographic identities, found that models asked to stand in for identity groups both misportray and flatten them; inference-time techniques reduced but did not remove these harms (measured without a control group; preprint, revised February 2025)⁷.
• Simulations drift in their own direction. A 2026 German study turned 514 survey respondents into language-model agents and simulated 16 quarterly rounds of opinion on a 2035 phase-out of new combustion-engine cars⁸. The agents moved significantly towards the strongest support level rather than holding the survey distribution, German-language prompts produced more decisive upward movement than English ones, and the authors warned that outputs should not be read as an accurate reflection of real citizens' beliefs⁸.
• Validation is unsolved. Researchers proposing preconditions for societal-scale agent simulations name the difficulty of validating simulation outputs and the dual-use potential of accurate models of human behaviour as critical, understudied challenges⁹. They propose not simulating populations without their participation and not simulating without accountability⁹.
• Even rule-based models stop short of forecasting. The Bank of England housing model's authors state it is not intended to produce forecasts⁴.
• The evidence is narrow. The strongest evidence on grounded agents comes from one US sample⁵; the counter-evidence comes from the US and Germany⁷⁸. No public-sector deployment was found.

Reasoning

The challenge for this idea is that the simulated town is most needed exactly where it is least reliable. Unintended effects are, by definition, behaviours that nobody anticipated, and language-model agents learn from records of past behaviour. A genuinely new incentive may produce responses that appear nowhere in that record.

“If a simulated town cannot reproduce the effects of a policy that has already been evaluated, it has not earned the right to predict the effects of one that has not.”

The drift found in the German study points to a subtler risk. If agents tend to converge on approval, a simulation could make a policy look more acceptable than it is, and give a decision already taken an air of having been tested. The people most exposed would be those the agents represent worst: groups the identity study found to be flattened⁷, and households with little voice in how the simulation is set up.

Cost and incentives also cut against it. Building agents from two-hour interviews with a representative sample is expensive and raises consent and data protection questions. A ministry under time pressure might prefer cheap demographics-only agents, which the evidence suggests are less accurate and less even across groups⁵. The organisations that sell simulation tools would have an incentive to present outputs as more certain than they are.

Finally, simpler tools may do much of the job. Behavioural elasticities added to static models, small pilots, randomised trials and structured consultation with affected groups already reveal many unintended effects. The simulated town would earn its place only if it found something those methods missed.

Signs to watch

• Peer-reviewed publication of the study of agents built from 1,052 interviews, currently a preprint revised in June 2026⁵.
• Any published backtest in which an agent simulation is run against a past policy with a known, evaluated outcome, and its predictions are compared with what happened.
• Any government or public body publishing a simulation development and deployment report of the kind proposed in 2026⁹.
• Papers and follow-up work from the PoliSim workshop on language-model agent simulation for policy, held at the CHI 2026 conference in Barcelona on 16 April 2026¹⁰.
• Any release by the operators of EUROMOD or national tax-benefit models of modules that simulate behavioural responses alongside the static rules.