Appendix BCase study: small models in a coding harness
The score, in three groups
24 of 29 models passed. That does not mean small models can replace big ones: the job was small enough that nearly every model could write the code.
What differed was how each group handled the work around it.
Small cloud models
11 of 13 passed
0.3¢ to $0.077 per passing run. One saved working code in the wrong folder; one ran on a server not set up to offer tools.
A vendor's small or fast tier (Lite, Flash, Mini, Nano, Luna, Haiku, Fast), run on the vendor's servers.
Local, downloadable models
1 of 4 passed
The "free model from Hugging Face" case, and the one that struggled most as an agent. Two never used a tool, one saved its code in the wrong folder, and a frontier model's plan didn't rescue the smallest. Only the 30B Qwen Coder passed, in 50 seconds.
Free model weights you download and run on your own computer: self-hosted on a workstation, or on-device on a laptop. No per-token price; hardware and electricity aren't counted.
Frontier models
12 of 12 passed
$0.03 to $0.25 per passing run. Dependable here, which is expected on a task this small. It doesn't show they were needed.
Each vendor's full-size tier, older generations included.
Appendix B · page 2 of 7
Turn pages with the ← and → keys.