Benefit Fraud Risk Scores: Widely Used, Thinly Proven, Repeatedly Withdrawn

The use case

A benefit fraud risk score is a number that a tax or welfare agency attaches to a claim or a claimant, estimating how likely it is that a payment is wrong or fraudulent. Staff then check the highest-scoring cases first, either before money is paid or afterwards. The scoring model is usually trained on the outcomes of past investigations: it learns which claim characteristics were common in cases where investigators found a problem¹¹¹.

What counts as AI here. This paper treats as AI only models that learn their scoring rules from historical data, known as machine learning. The UK Department for Work and Pensions (DWP) describes its Universal Credit advances model as a supervised machine learning classifier, meaning a model trained on past cases already labelled as fraudulent or not². Sweden's Social Insurance Agency, Försäkringskassan, confirmed to its data protection regulator that it had used a machine-learning-based risk profile⁸. Amsterdam's pilot used an "explainable boosting machine", a type of machine learning model built so that its reasoning can be inspected¹¹.

What is not AI here. Many agencies use hand-written rules, data matching and fixed scorecards. Amsterdam and Leiden, for example, used a "Fraud Scorecard" with crude hand-set risk factors such as neighbourhood and gender, later discontinued¹¹. Large human review programmes do similar work: DWP's Targeted Case Review of existing Universal Credit claims, run by around 6,000 staff, produced estimated savings of £581 million to March 2025 (operator-estimated, reported by the auditor)³. Those savings are not credited to AI in this paper. For the French family benefits score and the US tax audit selection algorithm, the sources consulted describe a data-driven risk score but do not confirm the model type; they are included because they do the same job and show the same problems⁹¹².

Out of scope: identity verification, forged-document detection and payment-network fraud tools, which address different risks.

Where it is running

The UK advances model is the only fraud and error machine learning model DWP runs at scale in live service¹. The National Audit Office (NAO), the UK's independent public spending auditor, counted four more in development and testing in October 2025³. In the United States, the Internal Revenue Service uses an audit selection algorithm to choose tax returns for audit, including claims for the Earned Income Tax Credit, a refundable tax credit for low earners¹².

Older deployments also exist. The Dutch Tax Administration used nationality as an indicator in a childcare benefit risk model until October 2018⁵, and Rotterdam suspended a welfare fraud system after a 2023 investigation found it biased¹¹.

What the evidence shows

No randomised trial of benefit fraud scoring was found. The strongest effectiveness evidence is a single operator's comparison against a random control sample.

The UK advances model. DWP sends a random sample of requests the model scores as low risk, together with a random sample of high-risk ones, to human decision makers. Comparing the two, it finds that requests the model flags are 2.5 times as likely to be confirmed fraudulent as a random selection (measured against a random control sample; operator-reported, April 2025 to March 2026)¹. The NAO reported a ratio of around three for the previous year and savings of £4.4 million since May 2022 (operator-estimated, reported by the auditor, October 2025)³. DWP says the model contributed to more than halving the estimated scale of advances fraud, now put at £20 million to £90 million for 2025-26 on £700 million paid (operator estimate; the range covers fraud and error together, and the model is one of several controls)¹.

Payment delay. Legitimate advances flagged by the model and then approved were paid a median of one day later than those approved automatically, the same delay as for other fraud controls (operator-measured, 2025-26)¹.

Uneven accuracy in the UK. DWP compares how often each group is flagged with how often those flags are confirmed by a human decision maker. All figures below are operator-measured for April 2025 to March 2026¹.

• Non-UK nationals were 3.35 times as likely as UK nationals to be flagged, but their flags were no more often confirmed (relative likelihood 1.04, not statistically significant).
• Claimants aged 55 to 65 were 1.36 times as likely as 25 to 34 year olds to be flagged, yet their flags were confirmed only 0.58 times as often.
• Couples were flagged at the same rate as single claimants, but their flags were confirmed only 0.36 times as often.
• Among the protected characteristics in UK equality law, only age and nationality could be tested; data on the others were too incomplete.

DWP's own reading is that, for these groups, the model is not working as well as it should¹.

Amsterdam's pilot. The city's first model was nearly twice as likely to wrongly flag an applicant with a non-Western nationality as one with a Western nationality, and 14% more likely to wrongly flag men (operator testing in 2022, independently verified by journalists)¹¹. After the training data were reweighted, internal testing showed a 20% accuracy gain over caseworkers (operator testing)¹¹. In the live pilot the model flagged more applicants than caseworkers did rather than fewer, was no better at finding cases that warranted scrutiny, and its bias reversed direction, now falling on Dutch nationals and women (operator findings reported by journalists, 2023)¹¹. The city's own analysis also found its caseworkers biased, against Dutch nationals and women¹¹.

Sweden. Lighthouse Reports and Svenska Dagbladet, with reviews from academic experts, analysed data on more than 6,000 people flagged by the agency's model in 2017. They found women, migrants, low-income earners and people without a university degree were selected more often, and that innocent members of these groups were more often wrongly flagged (independent analysis of agency data; the data are more than three years old)⁷.

United States. A peer-reviewed study of IRS data found Black taxpayers audited at 2.9 to 4.7 times the rate of other taxpayers, despite race-blind selection, mainly because of audits of Earned Income Tax Credit claims (measured, observational study, published February 2025; underlying data older)¹². Aiming the algorithm at total under-reported tax rather than overclaimed credits would reduce the disparity and detect more non-compliance, at a higher cost per audit (modelled counterfactual)¹².

France and Denmark. No published effectiveness figures were found for either system. A 2025 internal CNAF study, obtained by the organisations suing it, is reported to recognise the score's discriminatory effects (claimant-reported)⁹.

What it takes: cost, integration, governance

Cost. Public cost figures are scarce. Amsterdam estimated it spent about €500,000 on Smart Check plus €35,000 for a consultancy contract, and warned the total was approximate (operator estimate, reported June 2025)¹¹. DWP has not published the cost of its model. For context, DWP has £6.7 billion of dedicated fraud and error funding from 2020-21 to 2028-29, covering all activity and not only AI (allocated budget)³.

Integration. The UK model receives claim data in real time through an interface call from the Universal Credit system and returns a probability that the request is fraudulent². It is built and run in-house, with no third party involved². In Denmark, the operator ATP worked with private suppliers including NNIT to develop its models, and Danish law permits extensive merging of public databases to feed them¹⁰.

Human oversight by design. DWP's safeguards include: a human always makes the decision; reviewers are not told the risk rating or that a referral came from the model; flagged requests are mixed with random control cases; a declined advance does not affect the main benefit claim; and protected characteristics other than age are not used in training¹². These choices limit the effect of the score but do not remove it. The flag still determines who receives scrutiny, even when a person makes every decision.

Transparency. DWP's public record withholds the model's architecture, training data and scale of use under freedom of information exemptions². Big Brother Watch reports that DWP went to court to keep details of the model's data risks secret (advocacy group, July 2025)⁴. Försäkringskassan rejected nearly all of dozens of freedom of information requests over three years⁷. CNAF published the source code of its current model on 15 January 2026, after earlier refusals⁹.

Rules. Under the EU AI Act, AI systems that public authorities use to assess eligibility for public assistance benefits, or to grant, reduce, revoke or reclaim them, are classed as high-risk¹³. On 29 June 2026 the Council gave final approval to an amendment moving the start of high-risk obligations for such stand-alone systems to 2 December 2027¹⁴. Amnesty International argues that Denmark's system functions as social scoring, which the Act bans; the Danish agency rejects this¹⁰. Data protection law already applies: the Dutch regulator fined the Tax Administration €2.75 million, in part for using nationality as a risk indicator⁵⁶.

Stopped, paused or challenged

Where it falls short

Thin proof of effectiveness. Only one deployment publishes a control-group comparison, and it is the operator's own, unreplicated, and covers one narrow product. Its ratio fell from around three to 2.5 between years¹³. Its estimated savings of £4.4 million since May 2022 are small beside £9.5 billion of estimated UK benefit overpayments in 2024-25 alone (overpayment figure covers all fraud and error, not only areas the model touches)³. Much of DWP's recent progress came from human case review, not the model³.

What "better than random" leaves out. The UK ratio measures how often flags are right. It says nothing about how much fraud the model misses, or what each pound saved costs. The "right answers" the model learns from are past investigators' judgements, so any bias in past investigations can be learned and repeated¹¹.

The burden falls unevenly. In the UK, Sweden, Amsterdam and the US, some groups were flagged more often without being more often right¹⁷¹¹¹². Those groups are often among the most vulnerable: migrants, older claimants, single parents, low earners.

Amsterdam followed the responsible-AI playbook: expert advice, bias testing, reweighting and a staged pilot. In live use the bias moved to different groups and the model did no better than caseworkers¹¹.

A human decision is not the same as no harm. Being investigated has costs of its own. Swedish parents flagged by the model were left without money for basic needs during investigations⁷, and Danish claimants described living in fear of investigation¹⁰. The UK design, which checks before payment and costs legitimate claimants a median of one day, appears gentler than investigations after payment, though that rests on operator data¹.

The fraud being targeted may be overstated. In Sweden, journalists found the agency's fraud estimates rested on assumptions that did not check whether errors were intentional, and few alleged fraud cases reached court⁷.

The human baseline is biased too. Amsterdam's return to caseworkers meant returning to a process its own analysis found biased¹¹. The honest comparison is the model against the current process, measured the same way, and few agencies publish both.

Geography. The evidence is almost entirely European, plus one US study. No deployment outside Europe and the United States with published evidence was found.

Phase: In wide use — why

National agencies in the UK, France and Denmark run fraud risk scoring at scale in live service¹⁹¹⁰. The hypothesis of wide use is confirmed. Wide use is not the same as proven value, though: effectiveness evidence is thin and operator-produced, and the trend in several countries is contraction, with Sweden, Amsterdam and the Dutch tax authority having withdrawn models or risk indicators⁵⁸¹¹.

What to watch next

• 2 December 2027: EU AI Act high-risk obligations apply to benefit eligibility systems¹⁴.
• The Conseil d'État ruling on the CNAF score, and whether it refers questions to the Court of Justice of the EU⁹.
• DWP deploying its retrained advances model and publishing the fairness results for it¹.
• Any of DWP's four further machine learning models moving into live service³.
• A formal answer from Danish authorities or the European Commission on whether Denmark's models count as social scoring¹⁰.

Questions to ask before adopting

• Has the model been tested against both random selection and the current human process, and by whom?
• For each group, does a higher flag rate come with a higher confirmed rate?
• What happens to a flagged person who has done nothing wrong: delay, document demands or suspended payments?
• Will affected people know a score was used, and can they challenge it?
• Which law permits the profiling, and has a regulator reviewed it?