Appendix BCase study: small models in a coding harness
Does a big model's plan help a small one?
A better plan can't fix a model that doesn't use its tools.
For two local models, a frontier model (Claude Sonnet 5) wrote the plan, and the local model did only the work.
| Local model | Its own plan | Sonnet 5's plan |
|---|---|---|
| qwen2.5-coder 1.5Bon-device, on a laptop | Fail0 tool calls · 197 s | Fail0 tool calls · 167 s |
| qwen3-coder 30Bself-hosted, on a workstation | Pass3 tool calls · 50 s | Pass3 tool calls · 35 s |
The 1.5B model made no tool calls either way. The 30B model passed both ways and finished faster with the plan, though that run skipped writing its own test file, which the brief asked for. The hidden grader passed it anyway.
Appendix B · page 5 of 7
Turn pages with the ← and → keys.