ULPT: tuning a soft prompt in two dimensions
Fine-tuning a modern language model is expensive. So a lot of recent work asks a cheaper question: instead of updating billions of weights, can we learn a small “nudge” that steers a frozen model toward our task?
Prompt tuning is one of the tidiest answers. Rather than writing a prompt in words, you prepend a handful of learned vectors — a “soft” prompt — to the input and train only those. The model stays frozen, and you store one small prompt per task instead of a whole model.
But there’s a hidden tax, and that tax is what ULPT (Wu et al., 2026) goes after.
The hidden tax: the model picks your dimension
Each soft-prompt vector has to live in the model’s hidden dimension — the width the model uses internally, often thousands of numbers. A 100-token soft prompt for a model of width d is 100 × d trainable values, and d grows with every generation of models.
You didn’t choose that width; the architecture did. And there’s good evidence that task adaptation doesn’t need it: fine-tuning problems tend to have a low intrinsic dimension. So the parameter budget is set by the model, not by how much steering the task actually needs — and the extra width may even invite overfitting on small datasets.
Vanilla prompt tuning trains the full width; ULPT trains a tiny vector per token and expands it with a frozen random matrix.
The idea: tune small, project up
ULPT’s move is simple: optimize the prompt in a tiny space — as few as two dimensions — and use a fixed random matrix to project it back up to the width the model expects.
For each soft-prompt position you learn a tiny vector z (think 2D). A frozen random matrix up-projects z into a full d-dimensional embedding. The matrix is drawn once and never trained, and you don’t even store it: one random seed regenerates it when the model loads.
There’s one more ingredient, and it turns out to be essential. Learned prompt embeddings vary a lot from one dimension to the next, far more than ordinary word embeddings do. A random projection on its own struggles to match that unevenness. So ULPT adds two learned full-width vectors, a shift and a scale, which are shared across all prompt positions and adjust each output dimension. Without them, training loss stays high; adding the shift fixes most of that, and the scale helps further.
The trainable total is n·r + 2d instead of n·d. On T5-base with 100 prompt tokens and r = 2, that’s 1.7K parameters versus 76.8K — about 98% fewer. (At that setting, most of the 1.7K is actually the shift and scale; the prompt itself is just 200 numbers.)
Only the little vectors (plus a shared shift and scale) are learned; the random projection and the model stay fixed.
Why does a random matrix work?
This is the part that feels like it shouldn’t work. There are two reasons it does, and the paper proves both.
Expressiveness. By the Johnson–Lindenstrauss lemma, a random projection preserves the pairwise distances between a set of points with high probability — and therefore their dot products, which is what attention computes. The dimension you need grows with the logarithm of the number of prompt tokens, not with the model’s width. So a low-dimensional space linked to the model by a random map can capture the relational structure of a full-width prompt.
Optimization. Under standard assumptions, gradient descent through a fixed random projection (with the learned shift and scale) still reaches the global optimum. The frozen matrix doesn’t get in the way of training.
We also checked the obvious alternative: freeze the small vectors and learn the projection. At the same parameter budget, that’s slightly worse, and at the smallest budget it isn’t even possible: learning the projection at r = 2 already brings the total to 3.1K parameters, nearly twice ULPT’s 1.7K.
How well does it work?
Across 21 language-understanding and QA tasks on T5 (GLUE, SuperGLUE, MRQA and four more), the extreme r = 2 setting keeps at least 97% of vanilla prompt tuning’s performance. As far as we know, it’s the first demonstration of prompt tuning working in a two-dimensional parameter space.
Give it a little more room and it pulls ahead. At r = 64 (7.9K parameters), ULPT has the best GLUE and SuperGLUE averages in our comparison (86.0 and 76.8). That puts it ahead of LoRA, which uses 3.8M parameters, and of prompt-tuning variants like DePT and DPT. DPT is the closest relative: it learns the projection matrix instead of freezing it, and at the same rank it uses 7× more parameters and scores lower.
The pattern holds on reasoning with decoder-only models. On GSM8K and MBPP with Llama 3.2 1B and 3B, ULPT at r = 64 uses about 5–7K parameters. That’s about 4× fewer than vanilla prompt tuning and 20–300× fewer than LoRA. It posts the best average on the 1B model and ties the best on 3B, which FourierFT reaches with about 4× more parameters. It also uses the least GPU memory of any method, and because the prompt is computed once and cached, it adds no inference overhead.
Spend your budget on length, not width
There’s a result here I find especially telling. Fix the parameter budget and ask how to spend it. You could buy a short prompt of full-width vectors, or a long prompt of low-dimensional ones. Across T5-small, -base and -large, on both MNLI and Natural Questions, the long low-dimensional prompt wins every time.
It’s an intuitive trade once you say it out loud. Every extra prompt token is another position the transformer attends over and computes through at every layer. Under a fixed budget, more positions to steer from matters more than more precision per position. Width was the expensive luxury; length is where the budget is better spent — and ULPT’s cheap-per-token design is what makes that length affordable.
The same trade-off shows up in reasoning
I don’t think this is a quirk of prompt tuning. A recent paper, Effective Reasoning Chains Reduce Intrinsic Dimensionality, reaches a strikingly similar place from the reasoning side. It measures a task’s intrinsic dimensionality — the minimum number of model dimensions needed to reach a target accuracy. It finds that effective reasoning strategies push that number down, and that lower intrinsic dimensionality goes with better generalization.
That rhymes with ULPT. There, a prompt with more positions, steered from a tiny space, beats one living in the model’s full width. Here, a better chain of reasoning steps lets the model solve the task with fewer effective dimensions. Different settings, same shape: spread the work over more steps, and what needs to be learned collapses onto a low-dimensional subspace. It hints that “low-dimensional but extended” may be a general property of how these models do useful computation.
Why I think this matters
If you want one model to behave differently for hundreds or thousands of users or tasks, the bottleneck isn’t really inference — it’s storing and managing all those adaptations. ULPT shrinks each one to a few thousand numbers plus a random seed, which makes “one model plus a huge library of cheap task tweaks” practical.
The scope matters, too. We haven’t tested ULPT on models with tens or hundreds of billions of parameters. At that scale I’d expect it to shine at lightweight customization — style, output format, per-user behavior — rather than at teaching the model genuinely new capabilities, which belongs to pre-training.
The deeper takeaway I keep coming back to: the dimensionality of what you tune doesn’t have to match the dimensionality of what the model uses. A fixed random map decouples the two, and once you see that, much of the usual parameter budget turns out to be optional.
The full method, proofs, ablations and per-task numbers are in the paper.
References
2026
-
Ultra-Low-Dimensional Prompt Tuning via Random ProjectionIn Conference of the European Chapter of the Association for Computational Linguistics