Appendix BCase study: small models in a coding harness
Where small models fit
Try a small cloud model on small, clearly specified jobs you can check. Be careful with local models as agents. Where frontier models earn their cost is still open.
Worth trying a small cloud model
Small, clearly specified jobs with a way to check the result, in a good harness. Here, small cloud models passed the same check as frontier ones for a fraction of a cent to a few cents, and were often as fast.
Jobs with the same shape, which this study did not test: an accountant's script to reconcile two exports, an HR checklist pulled from a policy, routine test code for a developer.
Be careful with local models as agents
Very small or locally run models working on their own. Some never used a tool, one saved in the wrong place, and two took about 9 minutes. A frontier model's plan didn't fix a model that can't use tools. Local models can still suit chat or private drafts; that wasn't tested here.
Still an open question
Where frontier models earn their cost. This task can't tell; a harder, multi-step one would. Before you switch, try your own task, check the results with something the model can't edit, and run it more than once.
Go deeper: the scoreboard and the data
One row per model, grouped by size. Models and prices as of October 2026; names change quickly. Prices are the per-token prices the study recorded for the route each model was reached by: the vendor's list price, or a gateway or regional price, which can be about 10% higher.1
Every number in this case study comes from the study's own results, published below as CSV and JSON.2
Download the data (anonymized) CSV, every run · JSON copy
Every run, including the plan experiment and the setup runs left out of the scoreboard. Anonymized: the private gateway is relabeled "hosted model gateway", and host names, local paths and session details are removed. The run logs themselves aren't published.
| Model | Maker | Reached via | Result | $/M tokens | Cost | Seconds | Tool calls |
|---|---|---|---|---|---|---|---|
| Small cloud models 11 of 13 passed | |||||||
| claude-haiku-4-5 | Anthropic | hosted model gateway | Pass | 1.00 | $0.03 | 21 | 3 |
| DeepSeek-V4-Flash | DeepSeek | hosted model gateway | Pass | 0.21 | $0.011 | 36 | 12 |
| gemini-3.8-flash | Google API, direct | Pass | 0.75 | $0.077 | 31 | 8 | |
| gemini-3.5-flash | Google API, direct | Pass | 1.50 | $0.059 | 22 | 5 | |
| gemini-3.1-flash-lite | Google API, direct | Fail: wrong folder | 0.25 | $0.033 | 27 | 20 | |
| gemini-3.5-flash-lite | Google API, direct | Pass | 0.30 | $0.02 | 23 | 8 | |
| gpt-6-luna | OpenAI | OpenAI API, direct | Pass | 0.11 | $0.0039 | 41 | 13 |
| gpt-5.6-luna | OpenAI | hosted model gateway | Pass | 0.22 | $0.011 | 42 | 11 |
| gpt-5.4-nano | OpenAI | hosted model gateway | Pass | 0.22 | $0.014 | 53 | 18 |
| gpt-5.4-mini | OpenAI | hosted model gateway | Pass | 0.82 | $0.06 | 38 | 17 |
| grok-4-1-fast-reasoning | xAI | hosted model gateway | Pass | 0.22 | $0.004 | 117 | 3 |
| grok-4-1-fast-non-reasoning | xAI | hosted model gateway | Pass | 0.22 | $0.003 | 17 | 3 |
| Phi-4-mini-reasoning | Microsoft | hosted model gateway | Fail: no tools offered | 0.075 | – | 3 | 0 |
| Local, downloadable models 1 of 4 passed | |||||||
| gemma3:27b | self-hosted (Ollama) | Fail: no tool calls | – | – | 550 | 0 | |
| gpt-oss:20b | OpenAI | self-hosted (Ollama) | Fail: wrong folder | – | – | 529 | 6 |
| qwen2.5-coder 1.5B | Alibaba | on-device (Foundry Local) | Fail: no tool calls | – | – | 197 | 0 |
| qwen3-coder:30b | Alibaba | self-hosted (Ollama) | Pass | – | – | 50 | 3 |
| Frontier models 12 of 12 passed | |||||||
| claude-sonnet-5 | Anthropic | Anthropic API, direct | Pass | 2.00 | $0.063 | 21 | 3 |
| claude-sonnet-4-6 | Anthropic | hosted model gateway | Pass | 3.00 | $0.15 | 75 | 11 |
| claude-opus-4-6 | Anthropic | hosted model gateway | Pass | 5.00 | $0.15 | 41 | 5 |
| DeepSeek-V3.2 | DeepSeek | hosted model gateway | Pass | 0.58 | $0.25 | 82 | 20 |
| DeepSeek-V4-Pro | DeepSeek | hosted model gateway | Pass | 1.91 | $0.036 | 22 | 5 |
| gpt-5 | OpenAI | hosted model gateway | Pass | 1.38 | $0.097 | 83 | 11 |
| gpt-4.1 | OpenAI | hosted model gateway | Pass | 2.20 | $0.037 | 12 | 3 |
| gpt-6-sol | OpenAI | hosted model gateway | Pass | 2.20 | $0.03 | 34 | 4 |
| grok-4-20-non-reasoning | xAI | hosted model gateway | Pass | 1.38 | $0.21 | 91 | 36 |
| grok-4.3 | xAI | hosted model gateway | Pass | 1.38 | $0.049 | 30 | 13 |
| kimi-k2.6 | Moonshot | hosted model gateway | Pass | 0.95 | $0.045 | 21 | 5 |
| Mistral-Large-3 | Mistral | hosted model gateway | Pass | 0.55 | $0.03 | 20 | 5 |
Model names and prices are as of the study date (October 2026) and will change. Costs are estimates at the per-token price recorded for each route at the time. This study is not affiliated with any of the vendors named, and none of them reviewed this.
Notes (2)
- 1OpenAI, “Pricing,” OpenAI (developer docs), https://developers.openai.com/api/docs/pricing (accessed October 26, 2026).
- 2Steve Sparks, “Case study: small models in a coding harness,” AI, from the Parts Up, Appendix B, October 2026, https://aifromthepartsup.com/small-models/1 (accessed October 26, 2026).
Appendix B · page 7 of 7
Turn pages with the ← and → keys.