Lab notes

Lab notes · 04 · November 2026

When Fable reviews, Qwen fixes more

On 80 real issues from open-source projects, a published 4-bit fine-tune of Qwen3.8 27B, served on my own hardware, resolved 65 of the 78 that ran cleanly (83%). With Claude Fable 5.2 reviewing its first patches, for about $0.40 a task at API list prices, it resolved 74 (95%), fixing 9 and breaking none. In a follow-up, telling Qwen only that a patch was incomplete fixed 2 of the 11 real failures; Fable’s full review fixed 9. The gain needs a second run, and each run had twice the benchmark’s time limit, and Fable may have seen these public fixes in training.

By Robert Balas, Redcom Software

Tasks
80 of the 642 SWE-bench Pro V2 tasks, drawn at random within each repository before any run; hidden tests decide whether a patch resolves the issue
Worker
A published 4-bit fine-tune of Qwen3.8 27B (not trained by me) on a local DGX Spark, in a coding-agent harness, 99 minutes per run (the benchmark allows 49)
Helper
Claude Fable 5.2 in short read-only sessions: reads the issue, the repository and, for a review, Qwen’s patch; never sees the hidden tests; has no shell; may describe changes but not write them
Result
Qwen alone 65 of 78 (83%); with Fable’s review 74 of 78 (95%; 9 fixed, 0 broken, p = 0.004); with Fable’s brief 71 of 75 (95%; 89% if its 5 missing tasks count as failures); Qwen reviewing itself 63 of 78 (81%)
Cost
$0.40 per task for a review and $0.45 for a brief, at API list prices; the local hardware is not priced
Follow-up
41 of the tasks rerun with other help: the Fable brief repeated (8 fixed, 0 broken); Opus 5.5 and Sonnet 5.5 briefs did about as well at 50% and 18% of the cost; a verdict without Fable’s notes fixed 2 of 11 real failures, the full review 9
Status
Version 2: one run of the review arm, two of the brief arm (the second on 41 tasks); the run data is not public yet

The question

Open models that run on one machine can now resolve real issues in real repositories, slowly and not always. Frontier models score higher on public leaderboards, but every task sent to them costs money. A middle path is to let the local model do the work and ask the frontier model only for advice: a short brief before it starts, or a review of its first attempt. In this design the frontier model still reads the repository, so the middle path may lower cost, but it does not keep the code private.

I wanted to know whether that advice makes the local model solve more, and whether the gain comes from what the frontier model writes or simply from a second try. So I ran the same 80 tasks five ways, with two controls. The local model ran twice on its own, to measure how much a rerun changes by chance. It also reviewed its own patch, to separate a second look from a better reviewer.

A follow-up on 41 of the tasks then asked which part of the help matters and whether it has to come from Fable. It tried shorter briefs, briefs written by three other Claude models, and a review verdict with the reasons left out.

The setup

Tasks. 80 of the 642 tasks in SWE-bench Pro V2, drawn at random within each repository, with the seed fixed before any run. Each task is a real issue from one of 11 open-source projects, and a hidden test suite decides whether a patch resolves it.

Worker. A third-party fine-tune of Qwen3.8 27B, quantised to 4 bits. It runs on a DGX Spark I own (Nvidia’s desktop AI computer), served by sglang, an open-source model server. It works through dsh, a coding-agent harness that can drive any model, inside each task’s container. Harbor, a benchmark runner, starts the containers and grades the patches. The containers and the harness run on a separate client computer that calls the Spark over the network; two such computers were used in turn.

Helper. Claude Fable 5.2 in short, read-only sessions. It sees the issue text and the repository, and for a review also Qwen’s patch. It has no shell and no network, and it never sees the hidden tests or the reference fix. It is told to describe changes, not write them: no diffs, and no code block longer than three lines.

Arms. The same 80 tasks were run five ways, called arms (Figure 1). Each has a full name and a short one.

Arm Short name What Qwen gets
Qwen alone, pass 1 pass 1 The task only.
Qwen alone, pass 2 pass 2 The task only, run again with identical settings, to measure run-to-run noise.
Qwen reviews itself self-review Its own pass-1 patch on every task, with an instruction to check it against the issue and fix it. No Fable.
Qwen + Fable brief Fable brief The task plus Fable’s brief, about 500 words: where the problem is, its cause, the approach, the pitfalls and how to check the fix.
Qwen + Fable review Fable review Fable first reviews the pass-1 patch and answers APPROVE or REVISE. On REVISE, Qwen gets that patch again with the review attached. On APPROVE, nothing changes and the patch keeps its pass-1 result.

Time. Each run had 99 minutes of Qwen’s time, about twice the benchmark’s own 49, because the Spark slows down when it serves several runs at once. Container setup, Fable’s sessions and the final test run are not counted.

Terms. A task is resolved when its hidden tests pass. It is graded in an arm when at least one attempt there ran cleanly to the end; a few have no result in some arms (see Discarded attempts). Between two arms, a task is fixed when it goes from failing to passing and broken when it goes the other way.

Figure 1 · The five armsSame 80 tasks, five ways to run them. Red marks Fable’s part. Fable only advises: Qwen writes every patch, and the same hidden tests grade every arm.

What happened

Resolve rate

Arm Graded Resolved Rate 95% interval Within 49 min
Qwen alone, pass 1 78 65 83.3% 73.5–90.0% 50 (64.1%)
Qwen alone, pass 2 76 64 84.2% 74.4–90.7% 52 (68.4%)
Qwen reviews itself 78 63 80.8% 70.7–88.0% 40 (51.3%)
Qwen + Fable brief 75 71 94.7% 87.1–97.9% 60 (80.0%)
Qwen + Fable review 78 74 94.9% 87.5–98.0% 48 (61.5%)

The intervals are Wilson intervals. The last column applies the benchmark’s 49-minute limit to all of Qwen’s time on a task. For self-review and the review arm, that time includes the pass-1 run they start from. On that basis the brief arm still leads. The review arm falls just below pass 1 (61.5% against 64.1%): a review buys accuracy with a second run, and the second run takes time (Figure 2).

The same tasks, with and without help

Rates on slightly different sets of tasks can mislead, so each comparison below uses only the tasks graded in both arms. The p-value comes from an exact paired test (McNemar’s), which asks whether the tasks that changed went mostly one way. All four comparisons with Fable fall below the usual 0.05 threshold. Running six tests makes a chance result more likely, so with a Bonferroni correction (threshold 0.008) only the two review comparisons stay below it (Figure 3).

Comparison Tasks Resolved Fixed Broken p
Pass 1 → pass 2 (noise) 75 63 → 63 4 4 1.000
Pass 1 → self-review 78 65 → 63 1 3 0.625
Pass 1 → Fable brief 74 62 → 70 9 1 0.021
Pass 2 → Fable brief 73 62 → 69 8 1 0.039
Pass 1 → Fable review 78 65 → 74 9 0 0.004
Self-review → Fable review 78 63 → 74 11 0 < 0.001

Two identical solo passes disagreed on 8 of 75 tasks, 4 each way, so a shift of a few tasks is noise. Against that, the review arm fixed 9 and broke none. On the 72 tasks graded in every arm, the resolved counts are 61 in each solo pass, 59 for self-review, 68 with a brief and 69 with a review.

The brief’s evidence is weaker than its rate. Five tasks have no brief-arm result, because every attempt was discarded (see Discarded attempts). Pass 1 had resolved three of them. Counted as failures, they bring the brief arm to 71 of 80 (88.8%), and its comparison with pass 1 to 9 fixed and 4 broken (p = 0.267), which is no longer distinguishable from noise. One of the five was force-killed at the time limit on every attempt, which is close to a failure anyway. The review arm has no such gap: it has a result for every task that pass 1 has. A second brief run in the follow-up shows the same pattern (see Follow-up).

Figure 2 · Resolve rateBoth kinds of help lifted Qwen to about 95%; its own review did not. Share of graded tasks resolved in each arm, with the 95% interval. The hollow ring counts only tasks resolved within 49 minutes of Qwen’s time, the benchmark’s limit. For self-review and the review arm, that time includes the pass-1 run. The small tick on each row counts the tasks with no result as failures, out of all 80. The brief arm has 5 such tasks, which puts it at 71 of 80 (88.8%); the other arms have 2 to 4.
Figure 3 · Task by taskWith Fable’s review, Qwen fixed 9 tasks and broke none. For each pair of arms, on the tasks graded in both: tasks fixed (failing to passing) and broken (passing to failing), with the exact paired p-value. The shaded band is the noise seen between two identical solo passes, 4 tasks each way. Bold p-values pass a correction for six tests (below 0.008). The brief arm has no result for 5 tasks; counted as failures, pass 1 to Fable brief is 9 fixed and 4 broken (p = 0.267).

Was Fable’s review right?

Mostly, with many false alarms (Figure 4). Fable caught 11 of the 13 pass-1 patches that failed, and 46 of the 48 it approved did pass. But 19 of its 30 REVISE verdicts were on patches that already passed. Rerunning all 30 with the review attached fixed 9 and broke none, so here the false alarms cost time and nothing else.

What Fable wrote mattered more than the second try. On the 11 patches it marked REVISE that really failed, a plain second attempt resolved 3 of 10 graded, Qwen reviewing itself 1 of 11, a run with Fable’s brief 8 of 10, and a run with Fable’s review 9 of 11 (Figure 5). Self-review also did harm where there was little to fix: on the 48 patches Fable approved, 46 of which already passed, it broke 3 and fixed none. The review arm leaves approved patches alone, so it cannot break them. The follow-up tested whether the verdict alone would do as well; it did not (Figure 9).

Figure 4 · Fable’s verdictsFable caught 11 of 13 failing patches and sent back 19 that already passed. Each square is one graded pass-1 patch, grouped by Fable’s verdict and coloured by what the hidden tests later said.
Figure 5 · The real failuresA second try alone fixed 3 of 10; with Fable’s brief or review, Qwen fixed 8 or 9. The 11 tasks whose pass-1 patch Fable marked REVISE and that failed the tests, and how many of them each arm resolved. Each bar is out of the tasks that arm has a result for: pass 2 and the brief arm have none for one of the 11.

Follow-up: which part of the help matters

After the main runs, I ran 41 of the 80 tasks again with other kinds of help. The 41 are the 17 tasks Qwen failed alone in pass 1 or pass 2, 20 drawn at random (seed 20261115) from the 57 it solved both times that also had a brief-arm result, and the 5 with no brief-arm result, one of which is also among the 17. The set is harder than the full sample by design, so its rates cannot be compared with the 80-task rates above. The worker, the harness, the 99-minute limit and the rule for discarded attempts are the same as in the main runs.

Six kinds of brief were tested (Figure 6). Three come from Fable: its original briefs, given to a fresh Qwen run; a short note from a new Fable session limited to five tool calls and 150 words; and the original brief cut down to its first two parts, where and cause. The other three are full briefs from Claude Opus 5.5, Sonnet 5.5 and Haiku 5.5, written with the same instructions as Fable’s. A seventh arm tested the review. Qwen restarted from each of the 30 patches Fable had marked REVISE and was told only that a senior engineer had judged it incomplete, without Fable’s notes.

Each comparison below uses the tasks graded in both arms. The p-values are not corrected for multiple comparisons, and with 41 tasks only large differences can be told from noise.

Figure 6 · The follow-up briefsSix kinds of brief: three from Fable, three from other Claude models. Which of the full brief’s five parts each kind contains (where the problem is, its cause, the approach, the pitfalls and how to check the fix), and its median length. Red marks Fable’s writing, ink dots another model’s. The cut-down brief is Fable’s full brief with the last three parts removed.

The brief result repeated

The same Fable briefs, given to a fresh Qwen run, fixed 8 tasks and broke none against pass 1 (p = 0.008). The first brief run had fixed 9 and broken 1 on these tasks (Figure 7). The two brief runs agree on 33 of the 35 tasks graded in both. The repeat again has tasks with no result, 4 of them. Counted as failures, they turn its comparison with pass 1 into 8 fixed and 2 broken (p = 0.109). So in neither brief run does the gain survive counting the missing tasks as failures.

Less guidance helped less

The full brief has five parts: where, cause, approach, pitfalls and how to check. Both shorter versions keep only where and cause. Against pass 1, the short note fixed 8 and broke 2, and the cut-down brief fixed 7 and broke 4. Both fixed about as many tasks as the full brief and broke more that Qwen had solved alone. Against the full brief on the same tasks, the gaps are a few tasks and within noise (Figure 8).

Figure 7 · Follow-up against Qwen aloneFable’s brief repeated with 8 fixed and none broken; Sonnet’s did the same. For each kind of brief, tasks fixed and broken against pass 1 on the follow-up tasks graded in both, with the exact paired p-value, not corrected for multiple comparisons. The shaded band is the noise seen between the two solo passes on these tasks, 4 each way. The repeat run has no result for 4 tasks; counted as failures it is 8 fixed and 2 broken (p = 0.109). The first run has 5 (9 fixed and 4 broken, p = 0.267, counted the same way); the other kinds of brief have 1 to 4.

Briefs from other Claude models

Against pass 1, Opus 5.5 briefs fixed 7 and broke 1, Sonnet 5.5 briefs fixed 8 and broke none, and Haiku 5.5 briefs fixed 5 and broke 2 (Figure 7). On the tasks graded in both, the Fable brief resolved 32 to Opus’s 31, 32 to Sonnet’s 32 and 33 to Haiku’s 28 (Figure 8).

Opus and Sonnet briefs did about as well as Fable’s. On these 41 tasks a Fable brief cost $0.48 on average, an Opus brief $0.24 and a Sonnet brief $0.09, about 50% and 18% of Fable’s price. Haiku briefs cost $0.01 and helped less. Qwen resolved 28 tasks with them where it resolved 33 with Fable’s, the largest gap in the follow-up, though on 41 tasks it is still within noise (p = 0.125).

Figure 8 · Follow-up against Fable’s briefSonnet’s briefs did about as well as Fable’s at 18% of the cost; Haiku’s missed 6 tasks that Fable’s solved. Each kind of brief against the repeat run of Fable’s full brief ($0.48 per brief on these tasks), on the tasks graded in both: tasks only this brief solved (right) and tasks only Fable’s full brief solved (left), with the exact paired p-value. The first row is the same Fable briefs run twice, the noise to compare against. Under each name, the helper cost per brief at API list prices.

A verdict without the notes did no better than self-review

On the 30 tasks whose pass-1 patch Fable marked REVISE, Qwen was told only that its patch had been judged incomplete. It resolved 20 of the 30, one more than pass 1 and the same as when it reviewed itself. The bare verdict fixed 2 and broke 1. With Fable’s notes, the same tasks went from 20 to 28 resolved (p = 0.008). Of the 11 patches that had really failed, the bare verdict led to 2 fixes and the full review to 9 (Figure 9). The content of the review is what helps.

Figure 9 · Verdict or notesTold only that its patch was incomplete, Qwen fixed 2 of the 11 real failures; with Fable’s notes, 9. The 30 tasks whose pass-1 patch Fable marked REVISE, one square each, as each second attempt left them. The first group passed the tests in pass 1, the second failed. Every arm has a result for all 30. Squares are sorted within each row, so a column is not the same task from row to row. The upper line of each row is the 19 that passed in pass 1, the lower line the 11 that failed.

Cost

Arm Fable sessions Fable cost Per task Net tasks gained Cost per task gained
Qwen reviews itself 0 $0.00 $0.00 −2 –
Qwen + Fable brief 80 $36.08 $0.45 +8 $4.51
Qwen + Fable review 78 $31.30 $0.40 +9 $3.48

Fable’s cost is the API list price of each session, as reported by the session itself. Net tasks gained is fixed minus broken against pass 1, on the tasks graded in both. All 80 briefs are charged, including those for tasks with no brief-arm result. Counting those five tasks as failures, the brief’s net gain is +5, or $7.22 per task gained. Another $10.31 went on briefs and reviews thrown away after setup mistakes and is not in the table. Qwen ran on hardware I own, so its time is not priced: about 43 hours of graded runs for pass 1, 47 for pass 2, 37 for the brief arm, 22 for self-review and 21 for the 30 review reruns, the last two on top of pass 1. Discarded attempts are not counted.

Brief-arm runs averaged 29.5 minutes against 33.0 and 36.9 for the two solo passes. That is not evidence that a brief makes Qwen faster. The two identical solo passes differ by 3.9 minutes, about as much as the brief arm differs from pass 1, and the brief average leaves out its five tasks with no result.

The follow-up’s new briefs cost $10.14 for the 41 short notes, $9.80 for Opus 5.5, $3.57 for Sonnet 5.5 and $0.39 for Haiku 5.5. The repeat run reused the original Fable briefs, and the cut-down briefs needed no new session. Follow-up run times are not compared here: several arms shared the Spark at once, and the load changed from hour to hour.

Discarded attempts

An attempt is discarded when the harness itself fails: the connection to the model server drops, the harness crashes or has to be force-killed, or the starting patch cannot be applied. The rule is the same in every arm and does not look at the test result. Discarded attempts are retried, and a task with no clean attempt has no result. A run that reaches the time limit and exits cleanly counts as a result.

Reason Pass 1 Pass 2 Self-review Brief Review
Connection to the model server dropped 20 13 0 5 0
Harness crashed 3 2 0 2 0
Harness force-killed at the time limit 2 0 0 2 0
Harness exited with an error 0 1 0 1 0
Starting patch did not apply 0 0 3 0 6
Tasks left with no result 2 4 2 5 2

Self-review and the review arm have no result for the two tasks with no pass-1 patch to start from. All their discarded attempts were retried successfully.

The dropped connections were partly explained during the follow-up. The model server logged no error when they happened, and its proxy recorded them as closed by the client. The main client computer reached the model server over a VPN, and the VPN client rebinds its connections whenever Docker removes a finished task’s network. That timing matched one burst of four drops, not every drop, and it does not explain the drops seen with the other client computer. Separately, 23 runs made while one client computer was overloaded were set aside whatever their result, and those tasks were rerun on the other.

The follow-up arms, under the same rule:

Reason Fable repeat Short note Where and cause Opus 5.5 Sonnet 5.5 Haiku 5.5 Verdict only
Connection to the model server dropped 4 4 14 8 12 8 0
Harness crashed 2 2 4 2 2 4 0
Harness force-killed at the time limit 1 0 0 0 0 0 0
Harness exited with an error 1 0 0 0 0 1 0
Task setup failed or timed out 1 5 7 0 0 16 15
Tasks left with no result 4 2 2 4 3 1 0

Most of the setup failures came from too many tasks starting at once when an arm was launched. Some timed out during setup; I stopped others by hand within minutes to bring the load down (12 at the first launch, 10 at the second). All were retried.

What I take from it

  • Here, a short review from a stronger model did far more than a second attempt. On the 11 failing patches Fable sent back, its review led to 9 fixes; a plain retry managed 3 of 10 and a self-review 1 of 11.
  • Fable’s reviews raised many false alarms, 19 of 30 here. That is cheap only when, as in this design, a false alarm leads to a rerun rather than to a rejected change.
  • What helped was the content of the advice. Told only that its patch was incomplete, Qwen fixed 2 of the 11 real failures; given Fable’s review notes, it fixed 9.
  • The advisor need not be the most expensive model. On the follow-up tasks, Sonnet 5.5 briefs did about as well as Fable’s, at about a fifth of the price, on one run of 41 tasks. Haiku 5.5 briefs did worse, by a margin 41 tasks cannot separate from noise.
  • Cutting a brief to where and cause kept most of the fixes but broke more of the tasks Qwen could already do.
  • A gain measured with a generous time limit can vanish under a strict one. Under the benchmark’s 49 minutes the review arm gained nothing, and only the brief arm still came out ahead, by a margin not tested for significance.

What this does not show

  • Fable may know these fixes. The tasks come from public repositories, and Fable may have seen the real patches in training. On private code the gain could be smaller. This is the largest open question.
  • Qwen may know them too. The worker is a third-party fine-tune, and I do not know its training data. That would lift every arm, so it bears on the absolute rates more than on the comparison between arms.
  • The briefs are prescriptive. They name the files, functions and behaviour needed, so the result is closer to Fable diagnosing and Qwen implementing than to Qwen reasoning better.
  • The sample is small. 80 tasks, with one run of the review arm and two of the brief arm, the second on 41 tasks. The intervals are wide, and in neither brief run does the gain survive counting the missing tasks as failures. The follow-up comparisons between kinds of brief rest on 41 tasks and one run each.
  • The brief repeat reused Fable’s briefs. It shows that Qwen’s gain with the same briefs holds on a second run, not that a second set of Fable briefs would do as well.
  • The rates depend on the doubled time limit, and the review on a second run. Each run had 99 minutes, about twice the benchmark’s limit, and a reviewed task that was sent back got a second run on top of the first. At the benchmark’s 49 minutes the rates are 64.1% for Qwen alone in pass 1 (68.4% in pass 2), 80.0% with a brief and 61.5% with a review.
  • One worker, one harness, one benchmark. Published scores from other setups are not comparable with these numbers.
  • Fable was never run on these tasks alone, by choice. So this shows the pairing beats Qwen alone; it does not show how Fable alone would score, or how much cheaper the pairing is than using Fable for everything.

Next

This second version adds the follow-up, which asked which part of the help matters and whether it needs Fable. The two questions I most want answered are still open: whether the gain holds on fixes Fable cannot have seen in training, and how the pairing compares with Fable working alone.

The pattern is not new. Anthropic’s advisor tool, Aider’s architect mode and a memo from Fireworks in June 2026 all pair a cheaper worker with a stronger advisor. The Fireworks memo advised open-weight workers with Claude Opus 4.8 on three benchmarks, including 60 SWE-bench Pro tasks, where it reported gains of 4 to 7 points. Across its six benchmark-and-worker experiments, a review alone did as well as or better than a plan (their version of a brief) plus a review in five, and a reviewer of the same model gave no gain. This run also finds no gain from Qwen reviewing itself, nor from a bare verdict without the review’s notes, and finds the brief and the review about equal: on the 74 tasks graded in both, the review resolved 2 that the brief did not, and the brief 1 that the review did not (p = 1.000). Its gain over Qwen alone is larger, 11 to 12 points, but only under the doubled time limit. The worker, the advisor, the tasks and the time limit all differ, so the figures are not directly comparable.

Where the evidence is

Every run’s logs, patch, token use and test output are kept, as are Fable’s briefs and reviews with the cost of each session, the exact instructions Fable was given, the task lists, the follow-up briefs with the cost of each session, a log of what was started when during the follow-up, and the script that computes every number in this note from the graded runs. They are not public yet.