The Retirement Tax on API-Dependent Products

The cost that is not on the price list

Every product built on a commercial model API carries a liability that does not appear in the per-token price: the model will be retired, and someone will have to move the product to its successor. Two academic studies published in August and September 2026 give the first measured picture of how that work is actually done. Neither puts a money figure on it. Together they show that most teams do the work late, under pressure, and with tests that cannot see what has broken.

For a programme or portfolio owner, the practical consequence is simple. Model retirement is a recurring maintenance obligation with a date set by the vendor, not by you. It belongs in the plan and the budget as a known event, and its quality risk needs a different kind of test from the one most teams run.

What was studied

The larger study, by Hyungjin Lukas Kim of Myongji University, mined public GitHub history. From 22,555 commits in 17,703 non-fork repositories between 2024 and 2026, it matched 5,139 commits to official retirement events announced by OpenAI, Anthropic and Google¹. Two coders classified a 300-commit sample, with agreement of κ = 0.89–0.95 across five codes. Only 62% of matched commits were judged genuine migrations, and the headline figures are reweighted for that.

The second study, by Xiaonan Xu (Georgia Tech) and Wenjing Wu (University of Colorado Boulder), tested what happens to output quality when a product moves between successive models from one vendor. It queried 900 public benchmark items 50 times each on GPT-5.4, GPT-5.5 and GPT-5.6 Sol, across three benchmarks (SuperGPQA, Omni-MATH hard and IFBench), and counted an item as changed only if a Fisher exact test survived a 5% false-discovery control and the pass probability moved by at least 20 percentage points².

An earlier study by Wanqin Ma, Chenyang Yang and Christian Kästner, presented at CAIN 2024, ran the same kind of check on five GPT-3 and GPT-3.5 models for a single task, toxicity detection³.

All three are independent academic work. The two 2026 papers are preprints and have not been replicated.

Most teams migrate after the model has gone

Kim estimates that 82% of migrations (95% CI 79–84) landed after the old model had already been shut down¹. The unadjusted share was 72%. The median migration arrived 39 days after shutdown and 158 days after the announcement. In other words, the typical team had five months of warning and still moved only once the product had stopped working or been forced off the old model.

The pattern differs by vendor, and the difference tracks notice length:

The notice policies are the vendors' own published terms⁴⁵; the reactive shares are Kim's estimates¹. Google's events lacked usable notice dates and were excluded from the main regression.

At commit level, each e-fold increase in notice (roughly 2.7 times longer) cut the odds of post-shutdown migration by about 74–78% (p < 0.001)¹. OpenAI's Assistants API, retired on 26 August 2026 after a year's notice⁴, saw only 13–14% of migrations land after shutdown; the paper's abstract and its own table disagree on which.

The work itself was usually small. The median migration changed 2 files and added 14 lines; 78% were prompt-only; 94% edited source code because the model identifier was hard-coded. Fine-tuned products were the exception, at a median of 14 files and 693 lines. Only 8% of migrations switched provider¹.

Small is not the same as safe. In 14 of 180 human-coded genuine migrations (8%), the change failed silently. After three Anthropic retirements (Claude 3.5 Haiku, Claude 3 Haiku and Claude Opus 4.1), fix commits ran at 25 times the weekly volume of the preceding eight weeks¹.

The effort per migration is modest. The cost lies in when it happens: after shutdown, unplanned, and followed by a wave of fixes.

Notice length is the lever the evidence supports

The economic reading is that retirement cost is shaped less by engineering difficulty than by scheduling. A two-file change made in a planned window is cheap. The same change made after an outage, by whoever is on hand, followed by repair work, is not.

The evidence that longer notice causes earlier migration is suggestive rather than settled. The commit-level association is strong, but across the 13 retirement events with dated notice the correlation is ρ = −0.53 with p = 0.06¹. Teams that receive long notice may also differ from those that receive short notice. What the data does support firmly is the observation that matters for planning: with 60 days' notice, almost nine in ten open-source teams in the sample moved after the deadline.

A team cannot lengthen a vendor's notice period. It can treat the vendor's minimum as the planning horizon and start the work on announcement rather than on failure.

A better model is not a safe swap

The second cost is quality, and here aggregate benchmark scores mislead. Xu and Wu found that migrations whose average score rose by up to 7.3 percentage points still contained up to 8.3% of items that reliably got worse². Migrations whose average fell contained up to 10.7% of items that reliably improved. The averages hide movement in both directions.

The latest step in their test, from GPT-5.5 to GPT-5.6 Sol, lowered the Omni-MATH hard score from 53.0% to 48.9%². On IFBench, a 3.9-point regression under strict scoring shrank to 0.04 points under loose scoring, so the verdict on the same migration depended on how output was graded.

Ma and colleagues saw the same shape two years earlier with older models. Across their prompt and model combinations, 58.8% lost accuracy after an update, and 10.9% of individual predictions went from correct to wrong. In 87.9% of updates that improved overall accuracy, at least one previously correct prediction regressed³.

“An aggregate score can rise while the specific cases a product depends on get worse.”

For a product, that is the relevant failure. Customers do not experience the benchmark mean; they experience the specific requests the product was tuned to handle.

What the evidence does not show

The limits are material and should travel with any use of these figures.

• No price. None of the sources gives a cost in money or person-hours. Lines changed, fix-commit volume and silent-failure rates are proxies for effort and risk, not cost.
• Open-source only. Kim's sample is public GitHub repositories. Enterprise teams with release management, contracts and dedicated platform groups may behave differently, in either direction.
• Observation gaps. The commit search retained only the first 200–300 results per window, so counts are lower bounds, and migrations that leave no recognisable commit are missed¹.
• Narrow quality test. Xu and Wu tested one vendor's line, three benchmarks, and benchmarks chosen because the models still had headroom on them². Ma and colleagues tested one task on 2023-era models³.
• No replication. Both 2026 studies are single preprints. Some figures in Kim's paper are internally inconsistent, such as the Assistants API share.

These studies establish that retirement is a recurring, under-planned obligation and that aggregate scores are a poor acceptance test. They do not establish how much it costs, or that enterprise teams fare as badly.

What a decision-maker should conclude

Treat model retirement as a scheduled liability, not an incident. Each model a product depends on has a likely end date, and the vendor's minimum notice, 60 days at Anthropic and six months for OpenAI's generally available models, is the shortest planning window you will get. That date belongs in the roadmap and the run-cost budget at the point the model is adopted.

Two governance tests follow. First, can the team change the model without editing code? If the identifier is hard-coded, as in 94% of the migrations Kim observed, every retirement becomes a code release. Second, does the team hold a fixed set of its own critical cases, run repeatedly, against which a successor model is accepted or rejected item by item? If acceptance rests on an average score or a vendor benchmark, the evidence says regressions will pass unseen.

The honest position on cost is that it remains unmeasured. The useful position is that the cheapest migration is the one started on the day of the announcement and accepted against the cases the product actually serves.