Automated Essay Scoring: Widely Run, Operator-Validated, and Now Disputed by Districts

The use case

In spring 2024 the Texas Education Agency (TEA) reported that 9,690,388 written responses on its reading and language arts tests (known as STAAR) were scored by a hybrid system: software scores first, and people check a share⁴. This figure is operator-reported.

What counts as AI here. The engine uses natural language processing, a branch of machine learning that learns patterns in text. It works out how to score by analysing thousands of responses already scored by people³. TEA officials say it does not learn from one response to the next and is not generative AI of the kind behind chatbots¹³, although a 2024 news report noted it may use similar underlying technology².

What does not count. Human hand-scoring, the human share of hybrid scoring, and the human re-scoring done on request are not AI. Earlier engines such as the Educational Testing Service (ETS) e-rater, used for the GRE graduate admissions test, are reported separately by engine⁶. Student writing-feedback tools and AI-writing detectors are out of scope.

Where it is running

Texas moved quickly. TEA first mentioned automated scoring to a legislative committee in August 2022, announced hybrid scoring in fall 2023 and used it on a limited basis in December 2023². TEA said it had hired about 6,000 temporary scorers in 2023 and would need fewer than 2,000 in 2024 (operator-reported)². All Spanish-language tests remain human-scored because the engines do not work in other languages³.

The evidence is almost entirely from the United States, and mostly from one state. The 2019 survey is the latest multistate count found, and the ETS data are older still; no newer equivalents turned up. The one non-US decision found is a rejection, not a deployment.

What the evidence shows

No independent controlled evaluation of a deployed engine was found. The strongest evidence is the operators' own testing, followed by field outcomes reported by journalists from agency data.

Operator testing, 2023. TEA tested an engine on spring 2023 items. Models first built on field-test data did not hold up on live exams: field-test samples of 1,182 to 2,871 responses fell short of the 2,500 to 4,000 recommended, and TEA concluded that models must be reprogrammed on live responses during the scoring window (operator-reported)⁵. Once reprogrammed, TEA reports the models met its performance criteria for all students and for five groups: male, female, Black, Hispanic or Latino, and White students (operator-reported)⁵.

Operator testing, 2024. The pass marks were set against human scorers: the engine's agreement with the final score could not fall more than 0.10 below human-to-human agreement on a standard measure called quadratic weighted kappa, and average score differences could not exceed 0.15 standard deviations⁴. TEA reports that all items scored in spring 2024 met its full set of criteria on the full random sample (operator-reported)⁴. The reports' results tables could not be read in full, so no agreement rates are quoted here.

Why the engine hands off hard cases. Responses the engine scores with low confidence, below the 10th percentile, go to people⁴. A state document describes these as often on the border between two score points³. The engine is accurate where answers are clear and is routed away from where they are not.

Field outcomes, 2024 to 2026. Rescoring requests for reading open-ended answers rose from 2,758 in spring 2024 to 21,620 in 2025 and 64,489 in spring 2026 (agency data reported by the Texas Tribune)¹. More than 13,300 reading scores improved after review; of those, 42% had first been graded by people and 58% by the computer, and people did all the re-scores¹. Fewer than 0.5% of reading open-ended responses statewide changed¹. One district, Dallas ISD, resubmitted more than 5,100 tests and 34% returned with a higher score (district-reported)¹.

These numbers do not measure the engine's error rate. Districts choose which tests to resubmit, and one scholar attributes growth partly to awareness of the process, while another reads it as educators disagreeing with scores¹.

Cost. TEA estimates that scoring every response by hand would need four to five times as many scorers and cost an extra $15 million to $20 million a year (operator-reported projection)³²¹. This is avoided cost, not a measured saving.

What it takes: cost, integration, governance

Data and calibration. The engine needs a large pool of human-scored responses. A training sample of 3,000 responses double-scored by people was reported in 2024². In 2024 the model was programmed on 70% of the responses used, an ensemble step on 15% and evaluation on 15%⁴. Reprogramming on live human scores during scoring is now planned for all items, regardless of performance⁵.

Routing rules. About 10% of responses are routed at random for verification, with others routed for low confidence or for unusual conditions such as out-of-vocabulary text⁴. TEA states that an automated score is never used in the human scoring process⁵.

People. Even with the engine, a quarter of responses still reach a human scorer³. Costs also shift to schools: districts pay $50 per test if a rescore does not change the score, and rescoring is free if it does¹.

Rules. No law or regulator guidance specific to automated essay scoring appeared in the sources read. Governance is in practice set by the operator: TEA publishes study reports (publication dates not shown) and describes a hybrid model in which human experts drive the scoring¹.

Where it falls short

The challenge question is what would make the use case look less successful than it is presented. The answer starts with the design of the test.

Agreement with human scorers is the standard test, and it shows only that the engine copies the humans, not that the humans were right. Each acceptance test described compares the engine with human scores⁴⁶. The humans may be inconsistent, and a copy inherits that.

The evidence is not independent. The Texas studies are published by the agency that runs the programme, and the ETS evaluation was written by the engine's developer⁴⁵⁶. In February 2024 TEA said the engine was as accurate as human scorers but did not give the Dallas Morning News information showing it³.

A disputed spike in zeros. The state saw more zero scores on written answers in December 2023². A Lewisville superintendent said her district could not tell whether the cause was the test question or the new system². TEA said differences in the test-taking population, not hybrid scoring, drove the difference², and that a zero now also covers a response lacking coherent structure or evidence². The Dallas Morning News reported that roughly 8 in 10 English II written responses scored zero in fall 2023, against roughly a quarter in a human-only spring round (newspaper-reported, source not stated)³. The groups differ, so the cause remains unresolved.

Gaming and uneven effects. In 2019 Motherboard submitted two meaningless essays from a text generator to an ETS practice tool using e-rater, and each scored 4 out of 6 (a test of two essays)⁷. Motherboard also reported a December 2018 ETS study in which the engine raised scores of students from China by an average of 1.3 points on a six-point scale and lowered those of African American students by 0.81 points, compared with expert humans (second-hand report; the study was not opened)⁷. TEA's 2023 study reports on five groups; the sources read do not report results for English learners or students with disabilities⁵.

Dependence on people. The engine is calibrated to human scores, retuned during scoring and backed by human review of about a quarter of responses⁴⁵. Much of the system's reliability comes from people and process, not from the engine alone.

A rejection. Australia's education ministers decided in December 2017 not to use automated scoring for NAPLAN writing, after a New South Wales minister argued machines cannot judge interpretation or subtlety. Any change would need further research and a ministerial decision⁸. This was a decision made in 2017 and confirmed in 2018, so it is older than three years.

A different technology. A 2025 preprint tested five large language models on 67 Italian university essays and found human-model agreement low and not statistically significant, with repeat scoring unstable⁹. It is a small, unreviewed study of generative AI in higher education, not of the Texas engine.

No withdrawal of a US state programme and no lawsuit over automated essay scoring appeared in the sources read.

Phase: In wide use — why

The hypothesis, In wide use, is kept. A confirmed machine-learning engine scored a large share of 9,690,388 responses in one US state in 2024 (operator-reported)⁴, and a 2019 survey counted at least 21 states using engines in some role⁷. The survey did not establish which of those engines use machine learning, so the phase rests chiefly on Texas. It holds for English-language written responses in US state tests. Australia declined, and the evidence says little about other countries.

What to watch next

• TEA reading rescoring figures after the spring 2027 tests: whether requests continue to rise from 64,489¹.
• The next TEA scoring study reports: the sources read cover only 2023 and 2024, and TEA is examining whether to stop routing out-of-vocabulary responses to people⁴.
• Any independent audit of the Texas engine by subgroup, including English learners, which has not been found.
• Australia's Education Council: it said any change to the NAPLAN position would follow further research and be its own decision⁸.

Questions to ask before adopting

• What share of responses does the engine score alone, and what rule sends the rest to a person?
• Who ran the validation, on which student groups, and has anyone independent repeated it?
• How is the engine retuned against human scores during each scoring window, and who signs that off?
• What can a school or family do when they dispute a score, at what cost, and how often does it change the result?
• How does the engine treat fluent but meaningless or off-topic answers, and has it been tested on them?