steve sparks
16 of 42
ContentsPart II · Chapter 7

Chapter 7Small and free models

What small models are good for

Small, downloadable models are good tools for narrow, private or high-volume jobs. They know less and stumble more on long, open-ended work, so they complement the largest models rather than replace them.

Frontier models are the largest, most capable models, run in their makers' data centers. Small models are compact enough to run on a laptop or phone; one survey puts them at roughly 100 million to 5 billion parameters.1 Hugging Face, a site for sharing models, says it hosts over 2 million of them.2 Microsoft's Phi family is one well-known example, built to run on devices with or without a cloud connection.3

A good fit for

  • Private or offline work: once downloaded, they can run without an internet connection.34
  • Narrow, repetitive tasks such as tagging, extracting and routing.5
  • Working from text you hand them, where the facts are in the prompt.67

A poor fit for

  • Open-ended factual questions with no source in the prompt.78
  • Following strict, detailed instructions.9
  • Reliable tool use: one small model's own card warns it can make up function names.10
Notes (10)
  1. 1Zhenyan Lu, Xiang Li, Dongqi Cai, Rongjie Yi, Fangming Liu, Xiwen Zhang, Nicholas D. Lane, and Mengwei Xu, “Small Language Models: Survey, Measurements, and Insights” (preprint, arXiv, September 2024 (v3 2025-02-26)), https://doi.org/10.48550/arXiv.2409.15790.
  2. 2Hugging Face, “Hugging Face Hub documentation,” https://huggingface.co/docs/hub/index (accessed October 26, 2026).
  3. 3Microsoft, “Phi open models,” Microsoft Azure, https://azure.microsoft.com/products/phi (accessed October 26, 2026).
  4. 4LM Studio, “About LM Studio,” https://lmstudio.ai/docs/app (accessed October 26, 2026).
  5. 5Peter Belcak, Greg Heinrich, Shizhe Diao, Yonggan Fu, Xin Dong, Saurav Muralidharan, Yingyan Celine Lin, and Pavlo Molchanov, “Small Language Models are the Future of Agentic AI” (preprint, arXiv, June 2, 2025), https://doi.org/10.48550/arXiv.2506.02153.
  6. 6Patrick Lewis, Ethan Perez, Aleksandra Piktus, Fabio Petroni, Vladimir Karpukhin, Naman Goyal, Heinrich Küttler, et al., “Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks” (preprint, arXiv, May 22, 2020), https://doi.org/10.48550/arXiv.2005.11401.
  7. 7Marah Abdin, Jyoti Aneja, Hany Awadalla, Ahmed Awadallah, Ammar Ahmad Awan, Nguyen Bach, Amit Bahree, et al., “Phi-3 Technical Report: A Highly Capable Language Model Locally on Your Phone” (preprint, arXiv, April 2024 (v4 2024-08-30)), https://doi.org/10.48550/arXiv.2404.14219.
  8. 8Google, “Gemma 4 model card,” Google AI for Developers, https://ai.google.dev/gemma/docs/core/model_card_4 (accessed October 26, 2026).
  9. 9Marah Abdin, Jyoti Aneja, Harkirat Behl, Sébastien Bubeck, Ronen Eldan, Suriya Gunasekar, Michael Harrison, et al., “Phi-4 Technical Report” (preprint, arXiv, December 12, 2024), https://doi.org/10.48550/arXiv.2412.08905.
  10. 10Microsoft, “Phi-4-mini-instruct model card,” Hugging Face, February 2025, https://huggingface.co/microsoft/Phi-4-mini-instruct (accessed October 26, 2026).

Chapter 7 · page 1 of 2

Turn pages with the ← and → keys.

Cite this page

https://aifromthepartsup.com/small-and-free-models/1/

Using this book

© 2026 Steve Sparks. All rights reserved. You're welcome to quote short passages and cite any page, with a link back. To reuse diagrams or larger parts, ask through GitHub.

Quoted material and figures from other sources remain their owners'.