How to run a small language model on a laptop without a GPU

On the fourth of July, the open-source tracker at llm-stats still read “No open source releases this week,” with the freshest open-weight model, GLM-5.2, dating back to the middle of June. That is the third quiet week in a row. A quiet week is actually the best possible time to make this call, because it takes the shiny-new-model reflex off the table. The question a founder has to answer is smaller and more useful than the leaderboard: can a model that runs on a laptop, with no GPU and no cloud bill, do enough of the work to be worth it?

The reflex is to assume local AI starts with a GPU purchase order. It does not. It starts with picking the right small model, and most teams overshoot on size before they have run a single real example.

What a small language model actually is, and why one fits a laptop

A small language model is an open-weight model small enough to run on hardware already on hand, usually under about 10 billion parameters, and often in the 1B to 4B range for a machine with no discrete GPU. The trick that makes them fit is quantization: storing the weights at 4 bits instead of 16 shrinks the footprint by roughly four times, with a modest quality cost that a native 4-bit checkpoint keeps even smaller.

The footprints are genuinely small once quantized. Per a Digital Applied writeup from late June, Gemma 3 1B lands around 0.5GB at int4, Llama 3.2 1B near 0.7GB, Gemma 3 4B about 2.6GB, and Phi-4-mini roughly 3GB at Q4. Gemma 4 E2B loads in under 1GB in its mobile format. The sizing rule is boring and reliable: take the GGUF file size, add about 500MB of overhead, and that is the minimum RAM.

So an 8GB laptop with no GPU has real options in the 1B to 4B tier. A 16GB machine opens up the 7B to 9B tier, where quality climbs noticeably. That is the whole map. What a small language model gives up in raw capability, it buys back in fitting on the hardware in front of you.

Six steps to run a small language model with no GPU

1. Size the machine before the model

   RAM is the budget. Take the quantized file size, add about 500MB overhead, and confirm it leaves headroom for the rest of the operating system. On 8GB with no GPU, stay in the 1B to 4B range. On 16GB, a 7B to 9B model is comfortable.

2. Pick the size tier, not the leaderboard

   A model that fits with room to spare feels instant; one that fills RAM to the edge swaps to disk and crawls. Candidates in the small tier include Phi-4-mini, Gemma 3 4B, Qwen3-4B, Llama 3.2 3B, and SmolLM2. Pick for fit and task, not for the top benchmark row.

3. Choose the quant on purpose

   Q4_K_M is the sensible default for a laptop: about a quarter of the memory, most of the quality. Prefer a model that ships a native int4 or QAT checkpoint (Gemma publishes these) over a naive post-hoc 4-bit, because the native version holds quality better.

4. Pick the runner for the hardware

   Ollama, LM Studio, and llama.cpp share the same engine on a CPU, so choose for the interface, not the speed. On Apple Silicon, an MLX-backed runner is faster: the Ollama v0.31.1 notes (Releasebot, July 1) report Gemma 4 running nearly 90% faster there through multi-token prediction.

5. Run 30 to 50 real tasks

   Averages hide failures. Before trusting the model, run it against a few dozen examples of the one job that matters, whether that is extraction, classification, or tool-calling. This catches the quiet degradation that a public benchmark will never show.

6. Decide what stays local and what routes out

   The small model does not have to do everything. A sensible pattern keeps the bulk of work local and sends the hard tail to an API. Digital Applied notes that under small-model-first routing, 80 to 90% of agentic steps can stay on the local model.

Where small-model selection quietly goes wrong

The first trap is chasing the biggest model that technically fits. Nothing humbles a laptop faster than a 7B model wedged into 8GB of RAM, thrashing the disk while a 3B would have answered already. Headroom is a feature.

The second trap is more subtle, and it is a speed one. The word “laptop” hides two very different regimes. One has an accelerator, either Apple Silicon or an integrated or discrete GPU. The other is pure CPU. They are not close.

That 47.9 tokens per second is the accelerated regime, and it is a real result of quantization plus hardware acceleration. A pure-CPU laptop lands much lower. PromptQuorum, testing without a GPU, clocks Phi-4 Mini at about 12 tokens per second on an Intel i7-12700 at Q4_K_M, with Gemma 3 2B nearer 15. Both machines are “a laptop.” Knowing which one is on the desk is the difference between a snappy assistant and a frustrating one.

The third trap is assuming small means automatically private and automatically free. Running on the machine does keep the data on the machine, and it is dramatically cheaper per call. But the license still has terms worth reading, and the electricity and the setup time are not zero. Small is a real advantage. It is not a free lunch.

What good looks like, in footprints and tokens per second

Here is the map a team can actually plan against, drawn from the late-June benchmark writeups. Treat the CPU-only speeds as the floor and the accelerated ones as the ceiling.

On quality, these are not toys. The same Digital Applied writeup puts Phi-4-mini at 73% on MMLU and Qwen3-4B at 83.7 on MMLU-Redux, which is enough for a great deal of real extraction, drafting, and routing work. The economics are where the founder lens sharpens. Small local models run roughly 10 to 30 times cheaper per invocation than a 405B frontier call, using NVIDIA’s own figures cited in that writeup.

The honest framing is not local replaces the API. It is local handles the bulk, and the API handles the hard tail.

That reframing is what a quiet release week clarifies. When nothing new is shipping, the value is not in a fresh download. It is in wiring a proven small model into the 80% of work it can do on hardware already paid for.

What I would put on a laptop on Monday

If I had one afternoon and an 8GB machine with no GPU, I would start with a 3B to 4B model at Q4_K_M, Phi-4-mini or Gemma 3 4B, because they leave headroom and answer fast enough to feel useful. On a 16GB machine I would step up to a 7B to 9B and enjoy the quality bump. On Apple Silicon I would reach for an MLX-backed runner, since the July Ollama update made Gemma 4 nearly 90% faster there for the price of installing a new version. That is the kind of win a drought week actually delivers: no new model to learn, just the one already chosen running faster on the hardware you already own.

Then I would run my 30 examples before trusting any of it. The quiet weeks are not a lull to wait out. They are the calm you use to make the boring, durable choice well.