SOTA Watch

They show that one large vendor is monetising agents quickly.

A merge rate that cannot name its denominator cannot support a staffing or tooling decision.

Treat "fine-tune or prompt" as a question about the task, not the technology.

Until a buyer can specify the test, a block rate is a marketing number, not an assurance.

Treat the framework as a replaceable dependency and test it that way.

Both outcome claims are builder-reported, and neither states a baseline.

Quoting one figure as "the rate" imports a denominator the reader has not checked.

A falling unit price and a falling gross margin can both be true at once.

A fine-tuned small model has beaten a frontier model on one narrow task in one company's account.

A benchmark score is a property of the model and the grader together.

For Chinese open-weight models, expect most of that list to be yours.

They show that an automated pass is a claim about the grader, not about acceptance.

A 1% success rate is reassuring for a summariser and alarming for an agent that runs on every pull request in a public repository, because the attacker can try as often as they like.

Small is not the same as safe.

A rate is a property of a configuration on a date, not of a vendor or a model family.

Language is a variable in the test harness, and it moves scores by amounts comparable to the choice of model.

When a composite score allocates something scarce, the weights are the decision and the leaderboard is one input.

A taxonomy tells a team where to look; it does not tell a sponsor how often a failure will occur.

The only length a delivery team can rely on is the one it has tested on its own task.

The lag may be right; the report does not show the arithmetic.

Treat a leaderboard rank as a statement about the variant that was submitted, and confirm which one that was before it informs a selection.

If removing a vendor's public benchmark score would change which model you chose, that choice rests on evidence nobody in your organisation has checked.

Results this unstable are not evidence of a durable skill.

Leaders who treat it as a headcount lever are acting on numbers their own executives cannot reconcile.

Sandboxed describes a compute boundary, not a permission boundary, and the two must be designed and verified separately.

The question is no longer whether AI makes developers faster; it is whether the organisation can still tell.

On this evidence, experience mainly makes people approve faster.

It is an efficiency bought with forgetting, and the programme has to decide in advance what the agent is not allowed to forget.

A gate with no stated failure condition is a sign-off, not a test.

Retry overhead is one component of running cost, not a proxy for it.

Salesforce's most transferable lesson is here, not in the 76%.

It proves that a US licence, taken alone, is not a revenue forecast.

A gross benefit is a claim about what AI produced; it is not a statement about what AI cost.

On its own, the headline figure is evidence of what Salesforce is selling, not of what customers are getting.

The figures show an outsourcer protecting profit while its revenue line shrinks; they do not show why the revenue is shrinking.

A reported success deserves the scrutiny given to a failure.

Downloads measure what was fetched, not what was deployed.

Treat output volume as an input metric, not a delivery result.