GGUF vs safetensors: which model format to download before you run locally

On Monday the Ollama team shipped a release candidate whose entire headline was, and I am quoting the changelog, that it “hardens GGUF create flows and lint handling.” No new model. No benchmark. Just quieter plumbing for the step where a model file becomes something runnable. A day later the stable v0.31.2 landed with CUDA compatibility fixes and, of all things, UTF-8 file handling. Meanwhile llm-stats spent the week reporting “No open source releases this week,” with the freshest open-weight drop still Google’s DiffusionGemma from June 10.

So this is a good week to talk about the least glamorous decision in local inference. Before a model runs on real hardware, it exists as a file in a format, and the format that gets downloaded decides most of what happens next. GGUF or safetensors is not a contest between a better format and a worse one. It is a question about what the file is for.

What safetensors and GGUF are actually built to do

Safetensors is the storage and distribution format. Hugging Face built it as a safe replacement for Python pickle, which could execute arbitrary code the moment a file loaded. It stores weights at full training precision, memory-maps them from disk, and loads tensors without copying them. In April 2026 it joined the PyTorch Foundation and became the default checkpoint format on the Hub, so most models now arrive as safetensors first. It is what teams fine-tune, what they serve on a GPU with vLLM, and what they convert from.

GGUF is the other end of the pipeline. It is a single-file, quantized, inference-first format that llama.cpp, Ollama, and LM Studio read directly. One file carries the weights and all the metadata, quantized anywhere from 2-bit to 8-bit with k-quant variants in between. It is built for CPU and mixed CPU/GPU inference, which is exactly why it is the format that lets a laptop run something a datacenter card used to hold.

The mental model that saves the most grief: route by format, do not convert everything. A model is trained and stored in safetensors. From there it either gets served on a GPU straight from safetensors, or converted to GGUF for portable and CPU-friendly inference. Same weights, two destinations.

Where the download choice quietly goes wrong

The failure I see most often is not exotic. A team wants to run a mid-size model on a single 24GB card, finds the official repo, and pulls the safetensors weights, because that is what the model card links first. Those weights are full precision. A 70B model in fp16 is roughly 140GB. The card has 24. The download finishes, the load fails, and the afternoon is gone.

The mirror-image failure looks nothing like it. A team grabs the first GGUF quant they find, ships it, and a week later tool-calling is subtly broken. The file ran fine. The quantization level was just too aggressive for that particular task, and nobody tested the one workflow that mattered before it went out.

And then there is the conversion step itself, which is precisely the plumbing Ollama spent this week hardening. Turning safetensors into a good GGUF is a real operation with real failure modes: wrong metadata, a botched tokenizer, an importance matrix that never got passed. When the changelog for a tool this widely used still reads “hardens GGUF create flows,” that signals a conversion path common enough and fragile enough to keep patching. Nobody writes lint handling for a step that never breaks.

The size math, from a 7B laptop model to a 671B monster

This is where the two formats stop being abstract. A 7B model in safetensors fp16 is about 14GB. The same model as a GGUF quant is 2 to 7GB depending on how hard it is squeezed. Scale that up and it gets dramatic. DeepSeek-R1 at 671B parameters needs about 715GB of disk as full safetensors weights. As a dynamic 1.66-bit GGUF quant, one write-up measured it at 162GB. That is the difference between a model no single box can hold and one that fits on a well-specced workstation.

Size is not the only axis. Load behavior differs too, and here safetensors has a genuine edge that its zero-copy design earns honestly.

That speed comes from mapping tensors directly out of the file instead of deserializing everything up front. It is one more reason safetensors owns the storage-and-serving job while GGUF owns the run-it-anywhere job. Neither number makes one format better. They make each format better at a different thing.

Where MLX comes in if the hardware is a Mac

If the box in question is a Mac, a third format walks in: MLX, Apple’s own array-framework format. GGUF still runs on Apple Silicon through llama.cpp, and for a long stretch that was the default answer. But MLX is the native path, and the gap now shows up in shipping software rather than benchmarks. Ollama’s v0.31.1 earlier this month reported Gemma 4 running close to 90% faster on Apple Silicon using multi-token prediction powered by MLX, on by default and requiring no configuration.

GGUF versus MLX on a Mac is the same routing question in miniature. GGUF gives broad portability and the whole llama.cpp toolchain. MLX gives the most headroom on Apple hardware when the deployment is pinned to it. Most teams running a mixed fleet keep GGUF as the common denominator and reach for MLX only where a Mac is the production target, not the dev laptop.

The format is not the model. It is the container chosen for a specific box and a specific job.

A rule that survives the next release drought

Since nothing shipped this week, and GetDeploying’s tracker had H100 rentals sitting around $3.45 an hour across 45 providers with on-demand pricing up about 14% since last July, this is a stable-ground moment. No new model to chase, no leaderboard to react to. Which makes it a good time to write the rule down.

Pick safetensors when the model runs on a GPU at full or near-full precision, when serving through vLLM, or when fine-tuning is on the table. Pick GGUF when the target is CPU, Apple Silicon, mixed offload, or a single portable file, and accept that quantization is part of the deal. Then, whichever gets picked, run 30 to 50 real examples from the actual workload at the exact precision that will ship. The benchmark on the model card was measured on full-precision weights. What ends up deployed might be a 4-bit quant. Those are not the same model, and the gap shows up first in reasoning and tool-calling, long before it shows up in an average score.

That standing eval is the whole safety net. It is cheap, it is boring, and it catches the quiet degradation that no format label warns about.

What I would tell you over coffee

None of this needs a datacenter or a PhD. It needs the recognition that the file format is a decision, made once, early, and cheaply reversible when it goes wrong. Download the safetensors when the plan is a GPU and full precision. Convert to or download a GGUF when the plan is a laptop, a Mac, or a CPU. Test the quant that actually runs.

That is the entire discipline, and it fits on an index card. The week the release feed goes quiet is the best possible week to get the boring part right, because there is nothing shinier competing for the attention it deserves.