Lab notes

Redcom Software · lab

Lab notes

Research I run alongside client work, on local models, agents and evaluation. Each note gives the method and the numbers, including the results that went against me. Every number comes from recorded data; run summaries are available on request.

  1. November 2026 · Controlled experimentWhen Fable reviews, Qwen fixes moreOn 80 real issues from open-source projects, a published 4-bit fine-tune of Qwen3.8 27B, served on my own hardware, resolved 65 of the 78 that ran cleanly. When Claude Fable 5.2 reviewed each first patch and sent back the ones it doubted, Qwen resolved 74, fixing 9 and breaking none. A follow-up found that Sonnet 5.5 briefs did about as well as Fable’s at 18% of the cost. The gain needs a second run, and each run had twice the benchmark’s time limit, and Fable may have seen these public fixes in training.Agents · Local models · Evaluation
  2. November 2026 · MeasurementA local model counted the corrections. A hand count disagreed.A 27B model on the author’s own DGX Spark, behind masking and a sandbox, read 20,174 prompts from fourteen months of Claude Code history on one of the author’s machines overnight and counted one request in eight as a correction. A blind hand count by the author on a small 2026 sample found nearly one in four, and the model missed most of those; Claude Opus 5.5 did little better. The hand check stopped a number that may be too low from being reported unchecked. The hand count hints at a fall late in 2026 and the model’s rate edges down, though the samples are too small to be sure.Agents · Local models · Evaluation
  3. August 2026 · Build note · test buildWhen in doubt, the agent doesn’t payBefore an AI agent pays for an API call over x402 (pay-per-request inside HTTP), a gate I built checks the live price and payee against the service’s listing and the buyer’s rules, and never takes missing evidence as a yes. In fixture tests on 40 synthetic providers, 35 built to misbehave, every one whose live terms drifted from its listing was denied and all five clean controls were allowed; nothing is live yet.Agents · Payments · Go
  4. July 2026 · Stopped · negative resultSharper on one dataset, reversed on the nextA side network 0.2% the size of a frozen language model read four of its layers and made its answers more decisive than a same-size control did, at all three fresh seeds tested. On a second dataset the effect reversed and no change to generated text could be shown, so I stopped under a rule written before the deciding runs.Frozen models · Preregistered · Pythia-160M