What Is Kimi Delta Attention (KDA)? The Channel-Wise Gated Delta Rule Explained

Kimi Delta Attention (KDA) is a linear attention mechanism that refines the gated delta rule with fine-grained, per-channel gating, letting a compact recurrent memory hold long context cheaply. Moonshot AI introduced it in the Kimi Linear paper as the attention core behind Kimi K3, the company’s flagship model, where it replaces most of the standard full-attention layers a large language model would otherwise need. The result is a hybrid design that keeps quality on par with full attention while using a fraction of the memory.

That trade-off matters because attention memory, not raw compute, is often what caps how long a context window a model can practically serve. KDA attacks that bottleneck directly — and you can put it to work in a live chat over at bota.chat.

KDA in one sentence

KDA refines the gated delta rule with fine-grained, per-channel gating so a compact recurrent memory can hold long context cheaply. Instead of storing a growing list of past key-value pairs the way standard attention does, KDA compresses everything seen so far into a fixed-size state and updates it one token at a time. Each update decides, at the level of individual feature channels, what to keep and what to overwrite — a distinction that separates it from both plain linear attention and its immediate predecessor, Gated DeltaNet.

From quadratic attention to the delta rule

Standard softmax attention compares every token to every other token in a sequence, so its cost grows quadratically with sequence length — doubling the context window quadruples the work. Linear attention breaks that scaling: instead of an ever-growing table of past keys and values, it keeps a fixed-size state that updates as new tokens arrive, so cost grows linearly instead.

The delta rule as associative memory

The delta rule treats that fixed-size state as an associative memory — a lookup table compressed into a matrix. At each step, the model writes a new key-value association into the state and, critically, first removes whatever old association used to sit at that key before writing the new one. This “erase, then write” pattern is what “delta” refers to: the update is the difference between what should be there and what already is, not a blind accumulation of everything seen so far.

Why plain linear attention needed a forget mechanism

Without some way to forget, a fixed-size memory eventually saturates — old and new information overwrite each other in ways that destroy earlier context. A few mechanisms have tackled that problem, each adding a gate that decides how much of the old state to keep at each step:

• Gated Linear Attention (GLA) adds a forget gate to plain linear attention, without the delta rule’s erase-then-write update.
• Mamba2 uses a single scalar decay applied across a whole block of channels — simple and fast, but coarse.
• Gated DeltaNet (GDN) pairs the delta rule with a forget gate applied once per attention head.
• KDA takes GDN’s combination further, moving that gate down to individual feature channels.

GDN’s head-wise gate is the direct ancestor of KDA — the last stop before gating got fine enough to become channel-wise.

Channel-wise gating: what makes KDA different

GDN’s forget gate is head-wise. Its Gated DeltaNet architecture applies one scalar forgetting rate per attention head, so every feature dimension inside that head decays at the same speed regardless of what it represents.

KDA gives each channel its own forgetting rate. Rather than one decay value per head, KDA computes a separate gate for every feature channel within a head, so some channels hold onto information for many steps while neighboring channels in the same head decay quickly.

That precision comes from a Diagonal-Plus-Low-Rank update. KDA’s channel-wise gate is structured as a Diagonal-Plus-Low-Rank (DPLR) transition matrix — a diagonal term handles the per-channel decay while a low-rank term captures interactions across channels, keeping the update cheap despite the added granularity.

The added granularity is not free — it means more gate values per step — but it pays for itself precisely in workloads that depend on pulling one specific detail out of a long context rather than summarizing it broadly:

• Long-document question answering, where the answer depends on a single sentence buried far back in the input
• Multi-file code search, where a variable defined early in a repository must be recalled accurately much later
• Agent memory over long sessions, where a fact stated once needs to survive many subsequent turns
• Needle-in-a-haystack retrieval benchmarks such as RULER

A coarser gate loses precision first on exactly these tasks.

The hybrid: KDA + full attention in a 3:1 ratio

Kimi Linear, the architecture built around KDA, doesn’t replace full attention outright — it interleaves the two, with each block of layers split as follows:

• Three layers of KDA, handling the bulk of sequence processing with a fixed-size, cheaply updated state
• One layer of Multi-head Latent Attention (MLA), the same compressed-KV-cache attention variant Moonshot already relies on elsewhere in its models

That single MLA layer gives the model a full, uncompressed look back across the entire context whenever the ratio calls for it, while the surrounding KDA layers keep the per-token cost low. Because most layers use KDA’s fixed-size state instead of storing key-value pairs for every token, the architecture cuts KV cache usage by 75% compared to a model built entirely from full attention, and reaches up to 6x higher decoding throughput at a 1-million-token context length. On the RULER long-context benchmark at 128,000 tokens, the hybrid scored 84.3% against 81.3% for a full-attention baseline — evidence the memory savings didn’t cost retrieval accuracy.

Those figures come from Moonshot’s own technical report rather than an independent benchmark, worth keeping in mind when judging how directly they transfer to a different workload.

Kimi Linear outperforms full attention with a sizeable margin across all evaluated tasks, while reducing KV cache usage by up to 75% and delivering up to 6x decoding throughput at a 1M context length.

Kimi Team, “Kimi Linear: An Expressive, Efficient Attention Architecture”

Anyone judging whether a hybrid attention design is worth adopting for a given workload can run through the same checklist Moonshot’s own comparisons implicitly follow:

1. Check the context length that matters for the workload. Efficiency gains from linear attention compound as context grows, so short-prompt use cases see less benefit than million-token ones.
2. Look for a KV cache figure, not just a parameter count. Cache size, not total parameters, determines memory pressure at serving time.
3. Ask what fraction of layers still use full attention. A 3:1 ratio like Kimi Linear’s is a deliberate trade-off, not the only option.
4. Demand a like-for-like quality benchmark, not just a speed number. Throughput gains mean little if retrieval accuracy drops.
5. Confirm the gating mechanism’s granularity. Coarser gates are cheaper per step; finer ones cost more but tend to preserve more precise long-range recall.

Where KDA fits in the bigger model

KDA is an attention-layer decision, and it sits alongside a separate architectural choice about how a model routes computation across its parameters: sparse mixture of experts (MoE) routing. The two answer different questions about the same model:

• Attention (KDA’s job) — how the model remembers and retrieves context from earlier in the sequence
• MoE routing (a separate job) — how much of the model’s total parameter count activates for any given token

Because those are independent design axes, a model can combine an efficient attention mechanism like KDA with an efficient routing mechanism like MoE without either one constraining the other. The same hybrid ratio and channel-wise gating logic could, in principle, sit inside a dense model or a different MoE configuration entirely, which is why readers comparing open-weight versus open-source AI models will run into KDA again wherever a lab publishes both its weights and the architectural detail behind them.

FAQ