Lab notes

Lab notes · 03 · November 2026

A local model counted the corrections. A hand count disagreed.

A local model, sandboxed and fed only masked text, read 20,174 prompts from fourteen months of Claude Code history on one of the author’s machines in one night and counted one request in eight as a correction. A blind hand count by the author on a random sample of 2026 prompts found nearly one in four. The model missed most of them, and Claude Opus 5.5 fared little better. The hand check caught a number that would otherwise have been reported unchecked, and it may be too low. The hand count hints at a fall late in 2026 and the model’s rate edges down, though the samples are too small to be sure. It is one person’s record.

By Robert Balas, Redcom Software

Record
20,174 prompts from the Claude Code history on one of the author’s machines, since September 2025 (slash commands and very short entries left out), the usage tally, and the session transcripts still on disk
Reader
Qwen3.8 27B on a local DGX Spark: sandboxed, sensitive strings masked, no tools
Check
100 prompts (70 at random, 30 the model called corrections) labelled by the author without seeing its labels, and by Claude Opus 5.5
Corrections
On 44 requests in the random sample: hand count 10 (23%, 95% interval 13–37%); the model 6, 2 of them among the 10. The model on the whole history: 12.5% of 12,879 requests
Usage
About $100k at API list prices (a list-price equivalent, not the amount paid) in Claude Code’s usage tally, Dec 2025 to Sep 2026; transcripts on disk cover a smaller, later part
Status
One person’s record, one model run, checked by hand; the dashboard and scripts are private. No client names, code or data appear in this note

The question

Most of the code I write is now written by coding agents that I direct. Claude Code keeps a history file of every prompt typed into it. The record studied here is my own, from one machine: leaving out slash commands and very short entries, it held 20,174 prompts by November 2026, going back to September 2025. Codex was also in heavy use from May to October 2026; its prompts are not in that file. Alongside the history there are session transcripts, from May 2026 for Codex and August 2026 for Claude Code.

The first question was what this record says about working with coding agents: where the money goes, how often the agent has to be corrected, and whether that is improving. The second was whether a model can be trusted to read the record, because nobody reads twenty thousand prompts by hand.

The setup

Each prompt in the history file went to the model provider once, when it was typed. Sending fourteen months of them again in one batch is another matter: one place, next to file paths, tool output and whatever else gets pasted into a session. The choice here was to keep the whole record local, so the bulk reading ran on Qwen3.8 27B, on a DGX Spark I own.

Three guards sit around the job that sends text to the model (Figure 1).

  • Masking. Every piece of text passes through a filter that replaces sensitive-looking strings with [MASKED] before the model sees it, so nothing of that kind can be copied into the labels and summaries that a private dashboard stores and shows. The filter is tested before each run, and the run stops if a test fails.
  • A sandbox. The job runs inside a macOS sandbox profile that makes the home folder read-only, except one results folder, and blocks outbound connections except to the model server. The script refuses to start outside the profile.
  • No tools. The model receives text and returns JSON. It cannot open files, run commands or reach the network.

Over one night the model gave each of those prompts one of seven labels: task, question, correction, approval, pasted context, automated message or other. It also wrote a post-mortem of each of the 180 most expensive Claude Code sessions still on disk. Counts, durations and costs were computed by ordinary scripts, not by the model. Cost is tokens multiplied by API list prices on one day in November 2026, so it is what the calls would have cost on the API, not what was paid.

Nothing from the prompts themselves appears in this note. It reports counts, rates and costs only: no client names, no code and no client data.

Figure 1 · The setupThe bulk reading stayed on the author’s own hardware; only 100 masked excerpts left it. The bulk reading ran locally behind masking and a sandbox. Only the 100 check prompts, masked and cut short, went to a second reader in the cloud.

What the record says

The record is a subset

The files on this machine do not hold all the agent work done in the period. Claude Code also keeps a running usage tally, which covers 30 December 2025 to 21 September 2026 and comes to about $100k at list prices. That work ran on a flat-rate subscription, so the figure is a list-price equivalent, not the amount paid. Transcripts survive only for the later months: Codex from May 2026, worth $13.8k, and Claude Code from August 2026, worth $14.3k (Figure 2). The tally and the Claude Code transcripts overlap in August and September, so the three figures do not add up to a total.

Figure 2 · What the record coversThe usage tally starts seven months before the Claude Code transcripts do. Each bar spans the dates a source covers, with its value (prompts, or cost at API list prices, which is not the amount paid). The dashed line marks 2 August 2026, where the Claude Code transcripts begin; the tally is pale after it. The tally counts Claude Code use on this machine; the transcripts cover a later and smaller part of it.

Cost piles up in a few long sessions

When a session’s context fills up, the agent summarises it to make room. This is called compaction. The 180 costliest Claude Code sessions on disk came to $10.3k, and the 18 that were compacted five times or more account for $5.1k of that, or 49% (Figure 3). The 8 compacted fifteen times or more, all long-running sessions, cost $3.9k between them. Per prompt, though, the heavily compacted sessions cost about the same as those compacted once or never, $2.93 against $2.84. They cost more because they ran longer, a median of 53 prompts against 4. The post-mortems describe the agent re-reading its plan, its notes and its logs after each compaction to work out where it was, but the cost per prompt does not show that re-reading as extra spend.

Figure 3 · The 180 costliest sessionsOne session in ten took half the cost. Share of sessions and share of their $10.3k cost, by how often each session was compacted.

The model counts one request in eight as a correction

A script removed 3,466 requests that one agent wrote for another, recognised by their fixed openings. The model’s own labels removed 3,829 entries that were not requests: pasted context, approvals, automated messages and a few others. That leaves 12,879 tasks, questions and corrections typed by a person. The model labelled 1,612 of them, 12.5%, as corrections. A correction here is any request that redirects the agent’s previous step, from a broken build to a change of wording. It measures how much steering the work took, not how often the final result was wrong. By its own account the commonest were that the agent had broken something (362), left the work unfinished (324), got the look or wording wrong (290) or taken the wrong approach (270). In the months with fewer than 250 typed requests, its rate swung between 4% and 28%. From March to September 2026 it stayed between 10.4% and 14.5%, and in October, with 3,300 typed requests, it was 10.0% (Figure 4).

Figure 4 · Corrections by monthThe model’s rate settled; the hand count on a sample of January to September requests ran higher. Bars: share of typed requests the local model labelled as corrections, pale in months with fewer than 250 requests; November 2026 (N*) runs to the 11th. Ticks under the axis: public release dates of the Claude Opus and Fable models, and of the Codex models released while Codex was in use; they mark when a model became available, not when it was first used here. Bands: the hand-count rate on the blind sample described below, with its 95% interval.

Checking the reader

To see how far the labels could be trusted, I labelled 100 prompts myself without seeing the model’s labels: 70 drawn at random from the whole history, all of which happened to fall between January and November 2026, and 30 that the model had called corrections. Claude Opus 5.5 labelled the same 100 as a second reader. Only those 100 short, masked excerpts went to a cloud model for this.

All three readers saw each prompt masked, cut to 650 characters and tagged with a generic project label. The local model also saw the nine prompts next to it in the history, because it labelled them in batches of ten. I also saw the date, and I remember the work. No reader saw the agent’s message just before each prompt, and that message is often what makes a prompt a correction.

Of the 70 random prompts, 44 were hand-labelled as requests: tasks, questions and corrections. Figure 5 shows how each reader labelled those 44.

Figure 5 · The blind checkThe three readers rarely flagged the same requests. The 44 requests in the random sample, one dot each, sorted so that the ten hand-labelled corrections come first; each position is the same request in every row. A red dot is a correction that reader shares with the hand labels; a dot in the text colour is one the hand labels did not call a correction.

The readers flag different requests

Both models missed most of the hand-labelled corrections, and most of what they flagged was not one. Across all 100 prompts the local model called 36 corrections, and the hand labels agreed with 16 of them. On this small sample the hand-count rate was 23%, with a 95% interval of 13 to 37%. The model’s 12.5% for the whole history may be low, and its labels are too unreliable to use one prompt at a time.

The filtering held up

The model sorted requests from everything else well; its labels were the weak part. Of the 70 random prompts, it kept 41 as typed requests. By the hand labels, 36 of those were requests, 3 were approvals or pasted context and 2 were automated or other, and it left out 8 that the hand labels called requests. So the 12,879 is roughly the right set of prompts. The problem is the labels inside it.

Where the labels are weakest

Across all seven labels and all 70 random prompts, Opus agreed with the hand labels on 37 and the local model on 29. Cohen’s kappa, which corrects for agreement by chance, puts that at 0.41 for Opus and 0.29 for the local model, on a scale where 0 is chance and 1 is perfect. The usual reading calls those moderate and fair. On these 70 the local model never used the “automated message” label, which the hand labels gave to 14 of them. On the 48 prompts not hand-labelled automated or other (the 44 requests plus 2 approvals and 2 pastes), it matched the hand labels more often than Opus did, 29 times against 24.

The post-mortems ended in fixed-choice verdicts, and those were the weakest output: the model gave 154 of the 180 sessions the same middle verdict, “partial”, whatever had happened in them. Since the verdicts hardly ever varied, they carried almost no information, and the dashboard no longer shows them.

The hand-count rate may be falling

The hand labels hint that corrections have become a smaller share of requests. In the random 70, 9 of the 28 requests from January to September 2026 were corrections (32%), against 1 of the 16 from October and November (6%). The samples are too small to be sure, but they point the same way as the model’s monthly rates (the bands in Figure 4).

What to take from it

  • Do not report a number built on model labels before labelling a random sample by hand. Here the check turned “12.5% of requests are corrections” into “possibly more than that, nearly one in four on the 2026 sample, and possibly falling”.
  • End a long session at a milestone and leave a short state file for the next one. This is how I work now. A very long session is hard to review and hard to resume. The cost data above does not show that it saves money.
  • A 27B model on one desktop machine was good enough to sort twenty thousand prompts into requests and everything else, and not good enough to label them. For a private record that is a useful division of labour: let the local model do the sorting, and keep measurement for a hand check.
  • The correction rate appears to have fallen through 2026, while typed requests grew from 178 a month to 3,300. Three explanations fit, and this record cannot separate them: the models improved (nine Claude and three Codex releases fall inside the period, marked under Figure 4); the prompts improved with practice; or the mix of work shifted towards tasks that need less steering. A rate split by model, on the months where transcripts survive, would be the first test.

Limits

  • A subset. These are not all the projects or all the agent use in the period. They are what this machine kept: Claude Code’s prompt history and usage tally, and the transcripts still on disk, from May 2026 for Codex and August 2026 for Claude Code.
  • One person. The record is a subset of the author’s own work. It describes one way of working with agents, not how agents work in general.
  • Small check, and a noisy one. The blind sample is 70 random prompts, all from 2026, so it says nothing about how good the labels were in 2025, when the model’s rate was highest. A second hand-labelling of the same prompts four days later, done without seeing any model’s labels but knowing the first result and with the model’s label definitions to hand, agreed with the first on 44 of the 70 (kappa 0.54) and with Opus on 52 of the 70 (kappa 0.68). It put the correction rate at 19% (7 of 36 requests), against 23% the first time. On that second pass the model’s 12.5% was within the interval (10 to 35%). The figures in this note use the first pass, which was done blind.
  • The tools changed. The tally alone lists ten Claude models over nine months (release dates are marked under Figure 4), Codex took a share of the work from May to October 2026, and the volume of typed requests grew: 178 in October 2025 against 3,300 in October 2026. A falling correction rate may reflect better models as much as better prompts.
  • Task difficulty. Harder work brings both longer sessions and more corrections, and nothing here controls for how hard the work was.
  • List prices. Spend is tokens multiplied by API list prices on one day, not money paid. Some Codex models had no published list price, so their prices are estimates.