Field notes on AI — and a hand if you need one.

The write-up covers the harness, the sampling temperature and the seeds — the three things that make a benchmark reproducible and the three things nobody publishes. Worth reading as a template for your own internal evals, not for the leaderboard position.

General-purpose model obligations start applying, and the guidance finally distinguishes a model provider from a deployer. If you fine-tune someone else's weights and ship it internally, read the deployer section — most enterprise teams are in it and assume they aren't.

At the new per-token rate, stuffing 200k tokens of context costs less than the retrieval infrastructure it replaces for a lot of low-volume internal tools. It does not change the calculus at scale, which is where most of the argument actually lives.

Two years, a team of nine, and the number is public. The interesting part is the split: roughly a third on the platform, two thirds on data access and governance work that existed before any model was involved.

The task set overlapped with a public repository that appeared in pretraining data. Scores dropped several points on the cleaned set. A useful reminder that an agent benchmark is a software supply chain, not a measurement.

Money is moving from model training to model measurement. That tracks what I see in engagements: nobody is short of models, everybody is short of a defensible reason to pick one.

The piece is unfair in places and right in the main: a term that covers both a retry loop and a multi-hour autonomous session is not carrying information any more. Proposes a taxonomy by failure mode rather than capability, which is the more useful cut.

The interesting engineering detail is the fallback: it routes to a hosted model when the local one is not confident, and the confidence signal is a separate small classifier rather than the model's own logprobs.