TokMem: packing a whole procedure into a single token

If you use a language model for anything repetitive, you’ve probably built up a little collection of prompts. The careful instructions for formatting a meeting summary. The system message that makes the model answer like a patient tutor. The five-paragraph specification that turns it into a halfway-decent SQL generator.

Each of those is a procedure — a reusable recipe for getting the model to behave a certain way. And every time you use one, you paste the whole thing back into the context window, and the model re-reads it token by token before doing anything useful. The recipe never gets learned; it gets re-explained, over and over.

That re-explaining is the problem TokMem (Wu et al., 2026) sets out to fix.

The cost of explaining yourself every time

Prompting is a wonderfully flexible way to steer a model, but it has stubborn downsides.

First, it doesn’t get cheaper with reuse. A long procedure costs as much on the thousandth query as on the first, and because attention cost grows quadratically with context length, long instructions are expensive. They also eat into the context window the actual input needs.

Second, retrieval doesn’t really fix it. The standard workaround is to store procedures externally and retrieve the relevant one per query, as in RAG-style memory. But what comes back is still text that has to be pasted in and re-read, and the system is only as good as its retriever.

Third, prompts don’t compose cleanly. If a task needs three of your twenty procedures, you concatenate three blocks of instructions and hope they don’t step on each other.

On the left, a long prompt is re-sent to the model on every query; on the right, TokMem feeds a single memory token to a frozen model.

Prompting re-feeds the whole recipe each time; TokMem replaces it with one small token.

One token per procedure

TokMem’s idea is to stop re-explaining and start remembering. Each reusable procedure is compiled into a single trainable token — a memory token — that is added to the model’s vocabulary like any other special token. Training is ordinary next-token prediction on examples of the form query → memory token → response, and only the memory tokens’ embeddings are updated.

Here’s the part that makes it more than prompt tuning: the model chooses the token itself. Memory tokens share their embeddings with the output layer, so after reading a query the model can predict which memory token comes next, just as it predicts a word. Then it generates the response conditioned on that token. For a multi-step task it predicts the next memory token after finishing each segment — parse, then search, then format — and chains procedures without any external controller. If a query doesn’t match any stored procedure, the memory tokens simply don’t win the prediction, and the model answers in plain text.

So one token plays two roles:

  • It’s an index — selecting it says which procedure applies.
  • It’s a control signal — once in context, it steers generation toward that behavior.

And because it’s always one token, the overhead is constant, no matter how long or intricate the original instructions were.

A single memory token acts as both a procedure index and a generation control signal, feeding a frozen language model.

One memory token, two jobs: pick the procedure, then steer the generation.

Keeping the model frozen

The design choice I find most satisfying: the backbone language model doesn’t change. All the procedural knowledge lives in the memory tokens, not in the model’s weights. That separation buys two things:

  • You can keep adding procedures. Training a new memory token doesn’t touch existing ones, so knowledge isn’t overwritten the way it is when you keep fine-tuning shared weights.
  • No retraining the model. Each procedure is a tiny, isolated thing you train on its own, while the expensive backbone stays put.

Isolation alone isn’t quite enough, though. Newly trained tokens tend to end up with larger norms than older ones, and since routing is a softmax over memory tokens, the new ones can crowd out the old regardless of relevance. A one-line fix — rescaling each new token to the average norm of the existing bank — keeps routing balanced. With it, old procedures show only mild declines as new ones arrive. Without it, smaller models noticeably forget.

(One caveat for honesty: in the multi-step tool-use setting, we first give the backbone a short one-time LoRA adaptation on a separate set of tools, so it learns the format of interleaving memory tokens with responses. That adaptation is merged in, and the backbone is frozen again before any of the evaluated procedures are learned.)

How well does it work?

We tested two settings on Qwen 2.5 and Llama models from 0.5B to 8B parameters.

Atomic recall. Each of 1,000 Super-Natural Instructions tasks is its own procedure, and tasks arrive one after another, the way memories accumulate in practice. The key question is whether the model picks the right memory token. At 1,000 tasks, TokMem’s routing accuracy stays above 94% even on the 0.5B model, while the Sentence-BERT retriever used for RAG drops below 80%. That shows up in output quality: on Llama 3.1 8B, TokMem averages 67.0 ROUGE-L, against 50.9 for RAG and 64.7 for LoRA fine-tuning. RAG starts strong but degrades steadily as the library grows; fine-tuning forgets; replay buffers help but still fall short. TokMem is also data-efficient: it beats LoRA at every training-set size we tried, down to 10 examples per task.

Compositional recall. Here each of 50 tools from the APIGen function-calling dataset is a procedure, and each query needs up to four tool calls in sequence. On Llama 3.1 8B, TokMem trains 0.2M parameters versus LoRA’s 3.41M — about 17× fewer — and still edges it out, with 99.1 vs. 98.1 F1 on picking the right tools and 89.3 vs. 87.6 on generating the right arguments. In-context examples and RAG lag far behind on both. TokMem also handles queries with more tool calls than it saw during training.

A detail that matters: where the token goes

Classic prompt tuning puts its learned vectors before the input. TokMem puts the memory token after the query: query, then memory, then response. To isolate the effect of placement, we had both layouts memorize text the models had never seen. With a single token, the after-the-query layout reached 90% of its best perplexity roughly 30% sooner and ended at a lower perplexity. The gap narrows as you add more tokens, which suggests that placement matters most exactly when each token has to carry the most.

Why I think this matters

The mental model I keep coming back to: prompts treat a procedure as something you re-explain; TokMem treats it as something you remember. Once a behavior is worth keeping, it deserves a compact, reusable home — not a paragraph you re-paste into the context window forever.

If the idea keeps scaling, it points toward frozen models backed by growing libraries of single-token skills: modular, cheap to extend, routed by the model itself, and far less prone to the interference that makes continual learning so painful today.

The full details, the experiments, and the math I glossed over here are in the paper.

References

2026

  1. tokmem.png
    ICLR 2026
    TokMem: One-Token Procedural Memory for Large Language Models
    Zijun Wu, Yongchang Hao, and Lili Mou
    In International Conference on Learning Representations