What It Would Take for Agents to Modernise Legacy Code Under Human Review

The problem

Starting point. An earlier paper in this series found that most professional developers now use AI coding assistants, while the evidence on real productivity is mixed: a randomised trial found experienced open-source developers took 19% longer with AI tools, and company experiments found developers completing 26% more tasks¹. That paper rated the use case as in wide use and noted that few results capture quality after code ships¹. It concerned writing new code day to day. This paper adds a different problem: using agents (AI systems that carry out multi-step tasks, such as editing files and running tests) to read, translate and test ageing systems written in languages such as COBOL, where the new code must reproduce the old behaviour exactly.

The scale. In July 2025 the US Government Accountability Office (GAO) named the 11 federal systems most in need of modernisation, chosen from 69 submitted by 24 agencies. Eight of the 11 use outdated programming languages, four have unsupported hardware or software, and seven run with known cybersecurity vulnerabilities (audit report, 17 July 2025)². Both Treasury Department systems on the list run on COBOL, an older business programming language, and assembly language, for which GAO says the pool of people with the needed skills is dwindling².

The cost of standing still. The federal government spends over $100 billion on IT each year². An earlier GAO testimony put the cost of operating and maintaining ten critical legacy systems at approximately $337 million a year, a figure tied to GAO's 2019 selection and so older than three years (congressional testimony, May 2023)³. GAO warned that high maintenance costs could limit agencies' ability to modernise³. Progress has been slow: by February 2025, agencies had completed three of the ten modernisations on the 2019 list². These scale figures come from one country's federal government; no comparable audit from another country was found.

What already works

Research on checking translation. IBM Research authors describe, in a July 2025 preprint (a paper not yet peer-reviewed), an industrial system for checking machine translation of COBOL into Java. Its checks include syntax checks, semantic checks that match variable use, procedure calls and calls to middleware (software that connects programs to databases and transaction systems), and a check for invented code, known as hallucination⁴. The authors argue that if a program has ten paragraphs and each is translated with 90% accuracy, the chance of a fully correct program is only about 34%⁴. In plain terms, small error rates in each piece multiply across a whole program, so translation that looks accurate piece by piece can still be wrong as a system.

Benchmark results. An April 2026 preprint from university and industry authors reports that a research model trained specifically on COBOL reached up to 73.95% compilation success and a 49.33 "Pass@1" score on a COBOL-writing benchmark, against 41.8% and 16.4 for a general-purpose commercial model (authors' own benchmarks, April 2026)⁵. Pass@1 is the share of tasks solved by the first attempt. For COBOL-to-Java translation the research model reached 97.9% compilation success and up to 83.91 Pass@1. For the reverse direction, Java-to-COBOL, it reached 34.93 Pass@1 while general-purpose models scored near zero⁵. Translation in the direction modernisation needs is therefore the stronger result, but these are scores on test tasks, not on running systems.

An independent leaderboard. A public code-migration benchmark includes ten COBOL-to-Java repositories, a mix of open-source COBOL programs and synthetic systems built for the benchmark, graded by hidden tests, with checks that zero any submission that wraps the original program or hardcodes outputs (benchmark operator, updated 16 October 2026)⁶. Its top-listed score is 69.83%, but the page does not make clear whether that figure covers the COBOL split alone, and its overall score weights the non-COBOL split three times as heavily⁶. The figure should therefore be read as a sign that even the leading entries do not pass all hidden tests, not as a COBOL measurement.

Reported production use. Morgan Stanley told a trade publication that it has modernised more than 17 million lines of COBOL, Software AG Natural and PERL code into Java and Python using an in-house platform, saved more than one million hours of manual coding, and cut coding tasks that took a week to half a day, with "full human oversight and robust guardrails" (operator-reported, 28 July 2026, not independently verified)⁷. North Carolina's motor vehicle division has contracted for $84.8 million to replace five legacy COBOL systems; the article reports no results yet, and this paper did not establish how much AI is used in that project (reported by trade press, 2026)⁷.

The possibility

If the conditions below held, one way this could work is a staged programme in place of a single large rewrite. An agency or bank would keep the old system running while agents first read the code and write a plain-language description of what each part does, for subject experts to confirm. Agents would then propose translated code in small units and draft tests from recorded real transactions. The old and new systems would run side by side on the same inputs, a practice called parallel running, and people would review every difference before one function at a time moved to the new code. Named staff, not the agent, would sign off each step.

The role of AI. The parts that depend on AI (language models and agents) are reading and describing old code, proposing translations, drafting tests and flagging differences between old and new output. Much of the rest is ordinary systems work, money and policy: building test environments, moving data, replacing the middleware that connects programs to databases, security approval, contracts, funding and staff training. If the evidence is read plainly, AI may shrink one cost, the writing of code, while business scoping, proof of equivalent behaviour, data migration and organisational change remain. AI may then be a smaller share of a successful programme than the translation results suggest.

What would have to be true

1. Equivalence can be shown, not assumed. Tests built from real recorded transactions, including rare cases, would need to show that old and new systems give the same answers. Per-piece accuracy compounds across a program, as the IBM authors argue⁴.
2. Evidence moves from files to whole systems. The benchmarks found test tasks and a small set of repositories⁵⁶. Real systems share data definitions, call each other and depend on middleware, and a purpose-built evaluation of that scale would be needed.
3. Engineering foundations are in place. Google's DORA research found gains where teams had strong automated testing, mature version control and fast feedback, and little or no benefit where systems were tightly coupled and processes slow (self-reported survey, September 2025)¹⁰. Old systems are often the tightly coupled kind.
4. Human review works. Reviewers would need to understand the old business rules, yet GAO says the pool of people with COBOL skills is dwindling². Reviewers would also need to resist over-trusting the tool: in the earlier controlled trial, developers measured as 19% slower still believed AI had saved them 20% (controlled trial, 2025)⁹.
5. Plans, funding and accountability are documented. GAO found modernisation plans for nine of the 11 systems, with all required elements for only three, and says that until plans are fully documented, modernisation initiatives have an increased likelihood of cost overruns, schedule delays and overall project failure².
6. Cutover is tested early. Data migration and the switch to the new system would need testing well before the decision to go live, as the regulator's findings on a large bank migration show⁸.

What could stop it

What the evidence shows

Large migrations fail without AI. In April 2018 TSB, which had approximately 5.2 million customers, moved to a new platform. Only about 10% of roughly 900,000 attempted internet banking logins on 23 April 2018 succeeded, 225,492 complaints followed between 22 April 2018 and 7 April 2019, and the bank paid £32,705,762 in redress. The Financial Conduct Authority fined it £29.75 million (final notice, 20 December 2022, older than three years)⁸. The notice says testing concluded only the day before the decision to migrate was taken⁸. This was a conventional platform migration, relevant here as a documented failure of a large migration, not of AI translation.

Slow public-sector progress. Agencies had completed three of the ten modernisations on GAO's 2019 list by February 2025, and GAO's request that Congress consider requiring modernisation plans had seen no legislative action as of February 2026².

Thin and narrow measurement. The checking research reports no headline accuracy for whole programs, and its authors argue that per-piece errors compound⁴. The benchmark results come from the authors' own tests, and the authors list as limits that the tests work at file level and do not capture dependencies across files or shared data definitions (read from a machine-extracted portion of the paper)⁵. The public leaderboard's COBOL split has ten repositories⁶. No independent study of a translated production system was found.

Forecasts of failure and doubts from analysts. Gartner warned in June 2026 that more than 70% of mainframe migrations started this year would fail because leaders overestimate generative AI (projected by Gartner; reported by CIO Dive; the original was not opened)⁷. A Gartner analyst called legacy modernisation an unsolved business problem rather than an unsolved technical one, and said vendor tools tend to move customers onto the vendor's own stack⁷. An executive at a managed-services firm said AI makes it easier to stay on the mainframe⁷.

Reasoning

The challenge question is what would make this less likely, less useful or more harmful than it first appears. If translation becomes cheap but proof of equivalence stays expensive, the saving could be smaller than the translation headlines suggest, and the cost could move to testing and review. If the people who bear the risk are citizens or customers, as in the TSB case where hundreds of thousands complained, the pressure to cut testing time could be costly in a way the project budget does not show.

Incentives may also work against it. A supplier paid to migrate code could also be the party certifying that the migration is correct, which would leave nobody independent checking the work. Simpler, non-AI options may do the job in some cases: wrapping an old system behind modern interfaces, training new staff in the old language, or keeping the system and funding its maintenance. If agents mainly make old code easier to understand and document, the most useful result could be a better-understood legacy system, not a replaced one.

Signs to watch

• Legislative action on modernisation plans. GAO's request that Congress consider requiring major agencies to develop modernisation plans was still open as of February 2026; passage of such a requirement is observable².
• First independent results from a translated production system. Published audit or inspection findings on a system translated with agents, such as the one reported by Morgan Stanley or the North Carolina contract, would show whether behaviour was matched⁷.
• Outcomes of mainframe migrations started in 2026. These can be compared with Gartner's projection that more than 70% fail, once outcomes are reported; no date is known⁷.
• Whole-system benchmarks. A benchmark that tests multi-program COBOL systems with shared data, or growth beyond the ten-repository COBOL split on the public leaderboard, would address the main evidence gap⁶.
• Peer-reviewed versions of the two preprints. Publication after review would add scrutiny to the translation and checking results; no dates are known⁴⁵.
• GAO's next progress report on the 11 listed systems. The date is not known; it would show how many have moved from plan to completion².