--- title: "A local embedding model in a multi-agent workflow: the small model that keeps memory tidy" slug: local-embedding-model-agent-memory date: 2026-10-28 excerpt: "Most local-model talk is about the chat model. Inside a multi-agent workflow, a second, much smaller local model does quiet work: spotting when two memory lines say the same thing. Here is how that works in Cerevisor 4.0, where it fails, and the guard that catches it." featured_image: "https://bbtxujdxvidaghmhxkqs.supabase.co/storage/v1/object/public/generated-images/blog-1793776978601-local-embedding-model-agent-memory.webp" featured_image_alt: Six abstract memory sheets with short line patterns sit in two rows, with a small blue model cube beside a similar pair. author: Kinan Hamwi author_url: https://www.linkedin.com/in/kinanhamwi/ canonical_url: https://cerevisor.com/blog/local-embedding-model-agent-memory updated_at: 2026-11-04T07:22:58.866189+00:00 --- # A local embedding model in a multi-agent workflow: the small model that keeps memory tidy By the Cerevisor team. Product details checked against the source code of Cerevisor 4.0.0. TLDR In a multi-agent workflow, the local model that matters most for privacy is often not the chat model at all. It is a tiny embedding model that decides whether two memory lines mean the same thing. It can run on any laptop, but it cannot read numbers well, so it needs a plain word check beside it. In Cerevisor 4.0 that pairing is already in place, and the chat model doing memory work is a setting worth checking. On 21 October, Nvidia announced a fresh family of open retrieval models, and The Register’s Tobias Mann summed up the sizes in one line. The interesting part for anyone running agents on their own machine is how far the range stretches. > "Nvidia's latest embedding and reranking models, based on Nemotron 3.5, support embeddings of both images and text across 34 languages. This time around, the models are being offered in sizes ranging from one to eight billion parameters, with NVFP4-quantized versions for those running on new Blackwell-based accelerators." The Register, October 2026 An embedding model is a model that turns a piece of text into a long list of numbers, so that texts with similar meaning end up with similar lists. It does not write anything. It only measures closeness. Billion-parameter embedding models like these are built for searching large piles of documents. The one sitting inside our own app is far smaller, and it does a much humbler job. --- ## The local embedding model already inside Cerevisor Cerevisor 4.0 keeps memory as one line per fact: things about the person, things about the business, and a page for each area of work. Everything Cerevisor learns from a run arrives as a suggestion that waits for a click. That raises a boring but real problem. After twenty runs, the same fact gets suggested in five slightly different wordings, and a suggestions list full of near-copies is a list nobody reads. To catch those near-copies, the app uses all-MiniLM-L6-v2, an open embedding model, in its quantized ONNX build. Quantized means the model’s numbers are stored at lower precision so the file is smaller and faster, at a small cost in accuracy. The source describes it as about 23 MB, producing 384 numbers per line of text. It loads only on first use, is stored under the Cerevisor folder in the user’s home directory, and runs on the computer itself. ~23 MB the size the source gives for the on-device embedding model that matches memory lines That size is the point. A model this small needs no graphics card and no server. It is the one piece of the memory system that never needs a provider at all. The [same model](/blog/harness-tax-wrapper-security-score) has a second job: matching a request to the right skill when the wording differs from the skill’s own description. The setting for it lives under Settings as Semantic skill retrieval, with three choices: Auto (on when the model loads), Always on, and Off. Auto is the default. --- ## How two memory lines get matched: three checks in order When a new suggestion arrives, Cerevisor compares it with the kept lines of the same kind, using three checks, cheapest first. How Cerevisor 4.0 decides two memory lines are the same fact CheckWhat it comparesBar to count as a match Same textThe two lines in lower case, with punctuation stripped**Identical** Word overlapShare of meaningful words the two lines have in common0.7 or higher MeaningCloseness of the two lists of numbers from the embedding model0.86 or higher A similarity score here runs from 0 (nothing alike) to 1 (the same). The meaning bar is set high on purpose. The note in the source explains the reasoning plainly: merging two different beliefs by mistake silently destroys one of them, while missing a duplicate just leaves one extra card to reject. That asymmetry is worth copying into any agent memory design. When in doubt, show the person a duplicate rather than lose a fact. If the embedding model fails to load, for example on a machine with no internet on first start, the meaning check is simply skipped. Matching falls back to the first two checks, and nothing errors. Key Insight For agent memory, a wrong merge costs more than a missed duplicate. Set the matching bar high and let the person clear the leftovers. --- ## Where a small embedding model fails: numbers and “not” Here is the failure that only shows up once real memory lines go through it. The word-overlap check ignores words shorter than three characters, which is usually right, since short words are mostly “a”, “of” and “to”. That rule also drops single-digit numbers. So “Publish 3 articles a day.” and “Publish 5 articles a day.” come out as a perfect 1.0 match, and “Don’t keep answers brief.” scores 0.75 against “Keep answers brief.” Both pairs are recorded in the source as known cases. Embedding models have a gentler version of the same weakness. Two sentences that differ only in a number or a “not” sit very close together in meaning space, because almost every word is shared. Left alone, that would turn a correction into evidence for the old line. Someone changes their target from three posts a day to five, and the memory quietly decides they agreed with the old number. The fix is not a bigger model. Before the overlap and meaning checks run, Cerevisor applies a plain word rule with no model in it: if one line is negated and the other is not, or the two carry different numbers or dates, they are never treated as the same fact. That rule leans towards “different” by design. The worst case is one extra suggestion to reject. > A tiny embedding model is good at telling that two sentences are about the same thing, and bad at telling whether they agree. That pairing, a small model for meaning plus a dumb rule for facts, is the part I would steal for any self-hosted memory layer. It costs nothing to run and it closes the one gap that makes small embedding models dangerous for memory. --- ## The chat model behind memory work is a separate choice The embedding model only matches lines. Writing the suggestions in the first place takes a chat model, and that one is chosen differently. In the Provider Hub, the Background model setting covers “Memory, Operator, and other background work.” When it is left on “Same as default”, the default provider does that work. If the chosen model fails, background work tries the other switched-on chat providers in turn. So a workflow can run entirely on a local model while the memory suggestions after it are written somewhere else. For a team that wants nothing to leave the machine, the simple rule is: point Background model at the local model, and keep only local providers switched on. The guide to [OpenAI-compatible and local providers](https://cerevisor.com/docs/guides/providers/openai-compatible-and-local) covers connecting one, and the [memory and learning overview](https://cerevisor.com/docs/guides/memory-and-learning/overview) explains what each layer holds. --- ## Limits worth knowing about this setup - all-MiniLM-L6-v2 is an English-first model. We have not measured how well it matches memory lines written in other languages. - The 0.7 and 0.86 bars were tuned on paraphrased lines from Cerevisor’s own memory, not on a public benchmark. - The [release notes](/blog/cerevisor-v38-local-ai-agents-lasting-conversation) for 4.0 say plainly that how well a real model drafts and splits memory lines has not been tested yet. - The Nvidia figures above are a vendor announcement reported by one publication. ## Check the Background model before the next local run Open the Provider Hub and look at what Background model says. If it reads “Same as default” and the default is a cloud provider, then memory work for a local workflow is not local. Switching it takes one click, and it is the single setting that decides where memory suggestions are written. #### Sources - [Nvidia woos wary Europeans with more open weights models](https://www.theregister.com/ai-and-ml/2026/10/21/nvidia-woos-wary-europeans-with-more-open-weights-models/5304552) - The Register, 2026-10-21