A good model should not read every part of its history the same way. It should know what deserves detail now and what can remain compressed.
Long contexts are often discussed as a problem of memory.
It seems natural to assume that if a model can remember more tokens, it can read longer documents and work with more complex code. From a computational perspective, however, the problem looks slightly different.
Does a model need to read its entire past at the same resolution?
People do not reread every sentence of a long text with equal intensity. We examine the latest sentence carefully while keeping much earlier passages as a rough sense of context.
The standard attention mechanism in a Transformer addresses the problem much more literally. With N tokens, it computes nearly N times N relationships. It is powerful, but expensive.
So the central question in long-context modeling is not simply how to remember more.
It is what to read clearly and what to let become blurry.
The research presented in this talk was later released by Jinha Kim, Younghun Roh, and Jaeyeon Kim as Retrofitting Linear Attention into Diffusion Language Models. The accompanying LLaDA-Hybrid implementation is also publicly available.
The Problem with Long Context Is Resolution, Not Memory
In the simplest terms, attention assigns scores to relationships between tokens. The model extracts vectors called queries and keys from each token, compares them, and uses the results to form an attention matrix.
As the number of tokens grows, the number of relationships grows much faster. Doubling the token count produces roughly four times as many relationships to examine.
Language-model inference includes a prefill stage that reads the prompt and a decoding stage that generates new tokens. An autoregressive language model consults earlier tokens whenever it produces the next one. As the context grows, so does the history that each new token may need to reference.
To “remember” a long document ultimately means being able to consult many parts of it when needed. But computing every pairwise relationship in the same way makes the cost grow too quickly.
The important variable is therefore not just context length. It is resolution.
Should the model see its entire history with equal clarity? Or should it preserve detail only where it matters now and compress everything else?
Full attention preserves relationships densely. Linear attention compresses them into a fixed-size state.
Compression Is Not a Trick but a Necessary Choice
Sparse attention is one prominent attempt to ease this problem. It avoids revisiting every old token on every step. Earlier tokens can be grouped into blocks, with computation focused on selected blocks or on the most recent tokens.
Linear attention goes further. It folds the previous context into a state of fixed size. Rather than allowing attention cost to grow quadratically with the token count, it carries the past forward as an accumulated summary.
This is not free. Instead of computing every token-to-token relationship exactly, the model approximates them through a compressed representation of fixed dimensionality. It becomes faster, but information may disappear. The core question for linear attention is therefore not merely “how much faster is it?” but “how much less can it lose while becoming faster?”
This distinction matters. Making attention efficient is not just an engineering trick. It determines the form in which a model carries its past.
Diffusion Language Models Separate Past from Present
Most language models we use are autoregressive. They generate one token at a time from left to right. After producing “Once,” the model chooses the next token, then another, and continues the sentence.
A diffusion language model generates text differently.
Just as an image diffusion model starts from noise and constructs an image through repeated denoising, a text diffusion model repeatedly revises masks or uncertain token sequences. Instead of writing a sentence one token at a time, it places an unresolved block and refines it over several steps.
Planning an entire long sequence at once and filling in the details later remains difficult. Practical systems are usually closer to block-by-block generation: they denoise and commit the tokens in one block, then move to the next.
This creates an important distinction.
The active block is still changing. The identity of each token has not been finalized, and tokens within the block affect one another as they are refined. This region still needs detailed bidirectional attention.
Earlier blocks, by contrast, have already been committed. Their contents are fixed. From the active block’s perspective, they are context to consult, not objects still being revised.
There is less reason to process these two forms of context in exactly the same way.
The active block is live context. Previous blocks are settled context. One requires a close reading; the other may be compressible.
In block diffusion, the active block keeps changing while earlier blocks remain as committed context.
Block-Hybrid Attention Turns Resolution into Structure
Block-hybrid attention translates this distinction into a computational structure. Within the active block, it retains exact softmax attention because the tokens are unresolved and continue to influence one another. Previously committed blocks are instead compressed into a fixed-size recurrent state and accessed through linear attention.
That is what makes the design hybrid. It does not use a single attention mechanism everywhere. It gives the present and the past different forms of attention: the present remains sharp, while the past is compressed.
The paper does not pretrain this architecture from scratch. It retrofits a pretrained 16B-parameter LLaDA 2.1-mini model. Six of its 20 attention layers are replaced with block-hybrid attention, while the other 14 retain standard softmax attention.
Training proceeds in two stages. First, the feature maps and gates of each hybrid layer are trained to match the softmax-attention output of a frozen teacher. LoRA adapters are then attached, and the entire hybrid model is adapted using the masked diffusion objective. The first stage reduces approximation error within each replaced layer; the second corrects errors that accumulate when all modified layers operate together.
Instead of training a new model from scratch, the retrofit combines attention transfer with end-to-end adaptation.
A 1.7× Speedup, and the Questions That Remain
The results are more interesting than a simple claim that the model became faster.
Under SGLang continuous batching, LLaDA-Hybrid outperformed the original softmax teacher at every tested concurrency level. End-to-end decoding throughput was 1.50 to 1.73 times higher. At 128 concurrent requests, throughput increased from 2,310.2 to 3,994.4 tokens per second. Because earlier context is accessed through a fixed-size state, the model can also support more concurrent requests within the same memory budget.
Performance was broadly preserved across coding and mathematics benchmarks, but it was not identical. Scores declined modestly on tests such as HumanEval and CMATH, while MBPP+ improved. That is why efficiency results should be read as a profile of which capabilities moved and by how much, not as a single average score.
A fixed-size state keeps cross-block attention cost independent of the length of the committed context.
There are clear limitations. Only six of 20 layers are linearized, so the remaining layers still carry attention cost and KV memory that grow with sequence length. Evaluation is also limited to moderate generation lengths similar to those used during retrofitting. Whether the same quality holds for long-form generation and long-context reasoning remains an open question.
The result should therefore not be read as a declaration that the long-context problem has been solved. It is evidence that linear-attention efficiency can be added to a pretrained diffusion language model through a comparatively lightweight retrofit.
Fast Models Decide What to Let Become Blurry
Research on reducing attention is ultimately research on what a model can afford to see less clearly.
Reading everything exactly would be ideal, but it is too expensive. Compress too aggressively, and the model becomes fast but less capable. The important decision is where accuracy is essential and where a rougher representation is enough.
Sparse attention selects where to spend computation. Linear attention compresses the past into a fixed-size state. Diffusion language models shift generation from writing one token at a time to refining a block in parallel. Block-hybrid attention brings these ideas together at one point.
Read the active block closely. Read the past lightly.
The sentence is simple, but it expresses an important idea for inference. A language model does not need to see its entire history at the same resolution. A good model should distinguish the part that deserves attention now from the part that can remain in the background.
The path forward may lead less toward models that remember everything exactly, and more toward models that know what to examine and what to summarize.