Simplicity Bias in Attention
We usually ask whether a model can fit the data. When many different solutions fit equally well, which one does training find?
Ideally, we would like it to find the simplest one. But “simple” needs a definition. If \(C(f)\) measures the complexity of a function and \(\mathcal L_D(f)\) measures how well it fits a dataset, one way to write this preference is
First, find solutions that perform well. The tolerance \(\varepsilon\ge0\) allows equally good or nearly equally good fits. Among those solutions, prefer a simpler explanation. This is one way to express Occam’s razor: explain the observations with as little unnecessary structure as possible. It gives us a goal, but leaves open how training gets there.
We have left the measure of simplicity unspecified. One way to make it concrete is to think about compression: how much information does it take to describe an explanation and the data it explains?
This is the idea behind minimum description length (MDL). Imagine sending a dataset \(D\) to someone else. We first describe a candidate model, or hypothesis, \(h\). Writing \(L\) for description length in bits, this costs \(L(h)\). We then encode the data using that model, which costs another \(L(D\mid h)\) bits.
With the encoding rules fixed, two-part MDL chooses the hypothesis that gives the shortest total description:
A more complex model might describe the data more accurately, shortening the second term, while taking more bits to describe itself. MDL asks whether that improvement is worth the extra complexity.
MDL makes the preference for simple explanations explicit. Here, it motivates the question; our experiments measure task performance rather than description length. Can a preference for simplicity emerge from the way we parameterize and train the model?
This question has been studied extensively in neural networks. Valle-Pérez et al. study how different parameter settings map to different functions, finding that simpler functions can occupy much larger regions of parameter space. This concerns which functions are likely under parameter sampling; understanding what training finds also requires studying the optimization process.
Other work makes simplicity concrete in different ways. Rahaman et al. study a bias toward low-frequency functions, or patterns that vary more gradually with the input. Huh et al. find that deeper networks favor representations with lower effective rank, where most of the variation is concentrated along relatively few directions.
A useful lesson from this work is that a preference for simplicity does not require a small or restrictive model. We can give a network a rich space of possible solutions, even through overparameterization, and use its architecture, initialization, and training procedure to favor simpler ones. The goal is to preserve the flexibility to fit complex functions while guiding learning toward simple solutions when those are sufficient.
What about attention?
How does this idea carry over to attention? A model sees a context \(\mathcal C\), such as a sequence of input-output examples, and uses it to predict an output for a new input \(x\):
Here, \(\theta\) denotes the model’s trained weights, and \(\hat y\) is its prediction. These weights stay fixed during inference, but changing the context can change the function from \(x\) to \(\hat y\). The same model can therefore express different mappings without updating its weights.
We usually choose an attention function by hand. Do these choices introduce different biases that make each one better suited to certain tasks? If so, choosing a fixed form may be restrictive. This brings us back to the idea of giving a model a flexible parameterization, then guiding training toward simple solutions rather than restricting what it can represent from the start. Can we formulate attention more generally, so that it starts simple and learns the complexity the task requires?
Linear attention and softmax
Let \(K\) and \(V\) contain the \(n\) keys and values as rows, and let \(q\in\mathbb R^d\) be a query. Then
These equations use raw dot-product scores. For fixed keys and values, linear attention is then linear in the query. Softmax makes the weights depend nonlinearly on it. Our trained models use QK normalization and a learned score gain, as described below.
Below, we visualize a single attention operation in two dimensions. The grid shows what happens to a grid of possible queries. Linear attention preserves a linear geometry. Softmax bends the grid and keeps its output inside the values’ convex hull.
There is an easy temptation here: softmax looks more expressive, so perhaps it should always perform better. With suitable keys, values, and scales, it can even approximate a linear response locally. But representing a function and finding it through training are different questions. Let’s look at two tasks.
Two simple tasks
In-context linear regression
We adapt the in-context regression task of Ekin Akyürek et al.
For each sequence, we sample a new matrix \(W\in\mathbb R^{2\times2}\) and generate examples using \(y_i=Wx_i\). The model sees eight input-output pairs, followed by a new input:
Its task is to predict \(y_9=Wx_9\). We train with mean squared error, averaged over the two output coordinates. There is no observation noise.
Since \(W\) changes between sequences, the model has to use the examples in its context to predict the new output.
Lookup on an 8 × 8 grid
Imagine an 8 × 8 grid, with each location assigned either 0 or 1.
For each sequence, we generate fresh labels and show the model all 64 coordinate-label pairs in a random order. We then give it one of those coordinates again. Its task is to return the label that appeared with that coordinate.
In this example, a query for (1/7, 1/7) should return 0. In the next sequence, that same coordinate could have a different label, so the model has to retrieve the answer from the current context.
Each coordinate uses two numbers, with eight evenly spaced values from −1 to 1 on each axis. Every record has five fields: the two coordinates, two entries for its label, and a role flag. Label 0 is encoded as [1, 0], and label 1 as [0, 1].
The highlighted record is [1/7, 1/7, 1, 0, 1]. Its query is [1/7, 1/7, 0, 0, −1]: the label is withheld, and the final flag changes from 1 to −1. The complete input has 65 tokens, each containing five numbers.
The model outputs two scores, one for label 0 and one for label 1. We choose the label with the higher score. We train with cross-entropy and report the percentage of queries answered correctly.
Model and training
We use the same architecture for both tasks and all three attention types: \(N=6\) residual attention blocks of width 32, followed by an attention readout. The models have about 27,000 parameters.
Every attention operation uses L2-normalized queries and keys, \(\hat q\) and \(\hat k_i\), and scores \(s_i=g\,\hat q^\top\hat k_i\). Each block and the readout learns its own gain \(g=e^\gamma\), where \(\gamma\) starts at zero. This keeps the gain positive and initially equal to one.
For softmax attention, the gain is an inverse temperature: \(g=1/T\). Increasing it lowers the temperature and makes the attention weights more concentrated.
The blocks also contain RMSNorm and ReLU, so a network using linear attention is not globally linear.
The main comparisons use batch size 1,024, 10,000 updates, and seeds 0–2. We tune peak learning rate separately for each task and attention type. The rest of the optimizer settings are shared.
Here is what happens when we train linear and softmax attention. Move across the curves to inspect a validation point, or select a seed.
Linear wins one comparison; softmax wins the other. The interesting part is that the gaps survive LR tuning. With this architecture and training budget, the attention function changes what training finds. Can we let that choice become something the model learns?
A polynomial family
Instead of choosing the score transformation in advance, give it trainable coefficients. For each block and the readout, define a numerator and denominator polynomial:
The constant denominator coefficient is fixed at one. The attention operation is
Notice the location of the denominator: we sum \(H\) over the visible keys. This is not a pointwise kernel \(G(s)/H(s)\), and the numerator and denominator do not have to share coefficients. Its weights can be signed and need not sum to one.
Linear attention is an exact special case: \(G(s)=s\), \(H(s)=1\). A second useful setting is the truncated exponential
This gives an approximation to softmax, not exact softmax at finite degree. The approximation depends on the score range. In the next playground, switch between these presets, change the degree, or edit the coefficients. The truncated-exponential preset illustrates the unconstrained family; it is not necessarily allowed by the denominator constraint we use during training.
Polynomial attention itself is not a new idea. PolySketchFormer studies polynomial kernels and sketching, while TaylorShift develops efficient Taylor-softmax computations. Here we are interested in learning the response, starting from the linear member of the family. Our implementation evaluates the dense attention matrix; the experiments are about learned behavior, not a speedup claim.
Training polynomial attention
We start every attention layer from the linear member of the family:
This means \(a_1=1\), with every other numerator coefficient and every nonconstant denominator coefficient initialized to zero. The constant denominator coefficient \(b_0\) stays fixed at one. Each layer starts with the same operation as linear attention, while the additional terms are free to change during training.
Keeping the denominator away from zero
The attention output divides by \(\sum_i H(s_i)\). If this sum approaches zero, the output can become very large. We control it by limiting how far \(H(s)\) can move from its initial value of one.
QK normalization gives us a known score range. Since \(s=g\hat q^\top\hat k\), Cauchy–Schwarz gives \(|s|\le g\). After each optimizer update, using the updated gain \(g\), we compute
Here \(\rho\) (rho) is a fixed hyperparameter that controls how far the denominator polynomial can vary from one. We set \(\rho=0.5\) in every main polynomial run; it is not learned during training. In contrast, \(m\) is computed from the current coefficients and gain.
If \(m>\rho\), we multiply all eight nonconstant denominator coefficients by the same factor \(\rho/m\). Otherwise, we leave them unchanged. This ensures \(m\le\rho\), while keeping \(b_0=1\). The gain and numerator coefficients are unchanged by this rescaling.
For any score the layer can produce, the triangle inequality gives
So every \(H(s_i)\) lies between \(1/2\) and \(3/2\). Adding these terms over the \(n\) keys visible to a query gives
The denominator therefore stays positive and away from zero. This constraint restricts its variation; it does not restrict the numerator coefficients. We apply the same rule in every attention block and the readout, on both tasks. The full optimizer settings are in the appendix.
With this setup, we can add polynomial attention to the comparison.
Polynomial matches softmax on lookup and comes close to linear attention on regression. In both tasks, it starts from the same simple attention function. How much does that starting point matter?
Simplicity through initialization
Our main polynomial runs use no coefficient regularizer. Instead, we introduce a bias toward simplicity through initialization: start with linear attention, and let training change the additional coefficients. The denominator constraint is there for stability; it does not penalize higher-degree numerator terms.
The polynomial family can still express linear attention when we start elsewhere. But can training recover the same low error? To test this on linear regression, we replace the linear initialization with a truncated-softmax initialization:
We compare this with linear initialization using the same seeds 0–2, batch size 1,024, and training recipe, including a peak LR of 0.008. Both initializations are rerun on the same RTX 2080 Ti GPU.
We show two comparisons because the truncated-softmax initialization starts outside the original denominator bound. The left plot keeps the bound at 0.5, so its denominator coefficients are rescaled after the first update. The right plot uses a bound of 1.8 for both initializations, which admits the truncated-softmax initialization unchanged. This checks whether the result persists when both starting points satisfy the constraint.
Linear initialization finishes with lower error on every seed in both comparisons. The linear attention function is available in either case, but starting from the truncated exponential does not recover the same performance within this training budget. Expressibility alone is not enough: the starting point also changes what training finds.
This supports using a simple starting point in this setting. It does not tell us whether the runs reach different basins, or whether their final attention functions differ in simplicity. We compare performance at a fixed recipe and budget, without tuning the learning rate separately for each initialization.
Why compare two denominator bounds?
At gain one, the truncated-softmax denominator has coefficient budget \(\sum_{r=1}^{8}1/r!\approx1.7183\). It starts outside the main recipe’s bound of 0.5, so the projection rescales its denominator coefficients after the first update.
We therefore repeat both initializations with a bound of 1.8, which admits the truncated-softmax initialization unchanged. This wider bound no longer guarantees a positive denominator throughout training. All six runs finish with finite validation losses. The other settings are identical within each comparison: degree eight, no coefficient freezing, no coefficient regularization, and the same flat-then-cosine schedule.
Appendix: the experiments
The attention comparisons use the final main runs; the initialization comparison is a separate set of matched reruns. The results include 18 main runs, eight stabilization ablations, 270 LR sweep runs, and 12 initialization runs.
Task data, input encoding, and losses
Linear regression
For each sequence, the entries of \(W\in\mathbb R^{2\times2}\) are drawn independently from \(\mathcal N(0,1/2)\). We independently sample nine inputs \(x_i\) uniformly from \([-1,1]^2\), and set \(y_i=Wx_i\).
Each of the eight example tokens is \([x_{i,1},x_{i,2},y_{i,1},y_{i,2},1]\). The ninth token is \([x_{9,1},x_{9,2},0,0,0]\), withholding the query’s answer. The input has nine tokens with five fields each.
For a batch of \(B\) sequences, superscript \((b)\) identifies the sequence. Training and validation use mean squared error over both output coordinates:
For comparison, let \(U\) be a candidate regression matrix fitted to the \(n=8\) context examples. Its ordinary least-squares objective and a gradient-descent update with step size \(\eta\) are
Akyürek et al. show constructively that transformers can implement such updates. Our networks are trained on query prediction error; we do not assume they implement this particular algorithm.
Lookup
The coordinate values on each axis are \(-1+2j/7\) for \(j=0,\ldots,7\), stored in float32. Fractions in the figure denote the ideal grid coordinates. Every sequence contains all 64 coordinate pairs in shuffled order. Each coordinate receives an independent label \(y_i\sim\operatorname{Bernoulli}(1/2)\). We choose one of the 64 records uniformly as the query.
Writing the coordinate as \(x_i=(x_{i,1},x_{i,2})\), its record is \([x_{i,1},x_{i,2},1-y_i,y_i,1]\). The query is \([x_{j,1},x_{j,2},0,0,-1]\), and the target is \(y_j\). Thus every input has 65 tokens with five fields each.
For sequence \(b\), let \(t_b\in\{0,1\}\) be its target and \(z_b=(z_{b,0},z_{b,1})\) the model’s two scores. We use cross-entropy and report classification accuracy:
The side figure uses a sequence generated by this same data generator. Its records are arranged spatially only for illustration; the model receives shuffled numeric records.
Learning-rate selection
For each task/model pair, we try 15 peak LRs, \(10^{-3}2^{k/2}\) for \(k=-4,\ldots,10\), using seeds 0–2. We select the lowest mean final MSE for regression, and the highest mean final accuracy for lookup with cross-entropy breaking ties. We use one LR per task/model pair, shared across its seeds. The highlighted points are the selected settings.
These three seeds are used for both selection and the final reruns. The reruns verify reproducibility, not performance on held-out seeds.
Validation examples are fixed and shared within a task: 32,768 regression sequences or 8,192 lookup sequences per point. Fresh training sequences use separate RNG streams. Validation is measured before training and every 1,000 updates through update 10,000. Shading shows seed ranges, not confidence intervals.
Polynomial ablations
We rerun seed 2 on each task at its selected polynomial LR. “No extras” uses neither projection nor coefficient freezing. “Freeze only” holds the coefficients for the first 1,000 updates without projection. “Projection + freeze” adds that hold to the final projected recipe. The final arm adds degree-weighted coefficient regularization with \(\lambda=0.01\) to projection alone, with no freezing. We do not retune LR separately for these ablations.
Without the denominator projection, the lookup “no extras” run becomes nonfinite. We report that as a failure, not as a final accuracy. The freeze-only lookup run finishes at 61.73%; projection alone, projection plus freezing, and projection plus regularization all reach 100% for this seed.
The regularizer used in that arm is exactly
The linear numerator coefficient \(a_1\) is unpenalized and \(b_0=1\) is fixed. This is an optional ablation; \(\lambda=0\) in every main polynomial result above.
Full optimization settings
The final recipe uses no coefficient freezing and no coefficient regularizer. The polynomial coefficients have zero AdamW weight decay; the shared network parameters use weight decay 0.01. All three models use global gradient clipping at one and the same schedule shape: constant LR for 6,000 updates, then cosine decay to 10% over the remaining 4,000.
| Architecture | 5 → 32 embedding; six single-head causal residual attention blocks; one final-query attention readout; two outputs. RMSNorm ε = 10⁻⁶, ReLU after each residual block’s output projection; no separate FFN or positional embedding. |
|---|---|
| Parameters and score gains | Linear and softmax models have 27,079 parameters. Polynomial attention adds 119 coefficients, giving 27,198. All models have seven learned positive score gains, one per block and one in the readout. We learn an unrestricted scalar \(\gamma\) for each gain and set \(g=\exp(\gamma)\), with \(\gamma_0=0\), so every gain starts at one. |
| Optimizer | AdamW, β = (0.9, 0.999), ε = 10⁻⁸. Weight decay 0.01 on shared weights and learned gains; zero on polynomial coefficients. Global gradient norm clip 1. |
| Budget | Batch 1,024; 10,000 updates; seeds 0–2. Same schedule shape and data generator for each attention type within a task. |
| Schedule | Updates 1–6,000 at peak. For updates 6,001–10,000, t = (update − 6,001)/3,999 and LR = peak × [0.1 + 0.9(1 + cos(πt))/2]. |
| Polynomial | Degree 8 in every block and readout, separate numerator and denominator coefficients. Exact linear initialization, fixed b₀ = 1, no coefficient freezing or coefficient regularization in main runs. Gain-aware denominator projection bound 0.5 after each update. |
| Reproduction | The saved audit of the 18 main reruns reports exact matches to their reference model parameters, optimizer/RNG states, and metric journals on the tested H100 / Torch 2.8.0+cu128 setup. |
Related work
The simplicity framing above draws on Grünwald’s MDL tutorial, the simplicity bias of the parameter-function map, spectral bias, and low-rank simplicity bias. These concern different measures of simplicity and do not establish that training minimizes description length. In a more specialized setting, Musat connects minimum weight norms to Kolmogorov complexity for constant-precision looped neural networks. Connecting those minimum-norm constructions to the solutions an optimizer finds remains open. The in-context regression setting follows the line of work represented by Akyürek et al.
For attention, Katharopoulos et al. develop recurrent kernel-feature linear attention; our signed identity-feature baseline is a simpler, different normalization. Random Feature Attention approximates softmax through random features. PolySketchFormer and TaylorShift study polynomial/Taylor attention and its efficient evaluation. BASED studies the state-size/recall tradeoff. Our emphasis here is the relationship between the task, the attention parameterization, and its initialization.
There is also prior work on simplicity bias in attention itself. Zhang, Saxe, and Latham study saddle-to-saddle dynamics and the progressive recruitment of attention heads. Our head count is fixed; we vary the attention function and its initialization. On Learning the Transformer Kernel learns spectral feature distributions, and ReBased learns transformations of a quadratic kernel to improve in-context recall. These are close precedents for letting the attention kernel adapt. Our experiments use independent degree-eight numerator and denominator polynomials, initialized to the signed linear operator, and compare the learned behavior on two contrasting tasks.
Citation
@misc{pari2026simplicity,
title = {Simplicity Bias in Attention},
author = {Pari, Jyo and Duggal, Shivam and
Torroba-Hennigen, Lucas and Hanashiro, Rafael and
Agrawal, Pulkit and Kim, Yoon},
year = {2026},
month = nov
}