Rethinking Attention Sinks

In short

Some attention heads pour most of their attention onto the very first token, an “attention sink”. I asked why particular heads become sinks, using Qwen3’s grouped-query attention as a natural experiment, and found that the query pathway is a causal contributor to which heads sink.

Motivation

This project grew out of my own reading on how transformers work. I started with attention, the part that felt most unfamiliar: the mechanism where each token decides which previous tokens are worth “paying attention to”, using queries (what the token is looking for), keys (what earlier tokens offer) and values (the information they pass on).

Reading A Mathematical Framework for Transformer Circuits was enlightening: attention is the only component that moves information between positions, and it decomposes into a QK circuit that decides where to attend and an OV circuit that decides what to write. Later, I read Efficient Streaming Language Models with Attention Sinks, and wanted to stop reading about the phenomenon and see it for myself. I chose Qwen3-1.7B, a model small enough to run on my own computer, but large enough to show real attention sinks.

Schematic What a sink looks like. Each row is a token and each square is how much it attends to an earlier token. Almost every token sends most of its attention to the first one.

The gap I noticed

Once I could load the model and plot its attention, I realised the interesting question was not that a sink exists but why particular heads become sinks. Several mechanisms have been proposed (key-side biases, softmax-normalisation effects, architecture-specific circuits), but how head-specific sink specialisation arises in models with shared key/value representations is less understood.

Qwen3’s grouped-query attention turned out to be a natural experiment for exactly this. Within a group, several query heads share the same keys and values, so if two such heads sink differently, the difference has to come from their queries. That controlled structure let me ask a clear, sharp question I could answer, instead of a more abstract question about attention in general.

Why GQA helps Heads in one group read exactly the same keys and values, so any difference in sinking must come from the queries.

How I investigated it

I approached the sink from three angles.

(01) Cause

Query transplants

I exploited the shared key/value structure of GQA to isolate the query pathway, and tested whether fully or partially transplanting one head’s query transfers its sink behaviour to another.

(02) Depth

Layer-by-layer trajectories

I treated each head’s sink not as a fixed score but as a trajectory across depth, tracking how attention concentration, redistribution and predictability evolve layer by layer, with matched control heads throughout.

(03) Function

Full vs windowed attention

I perturbed sink attention under both full-context and windowed (streaming) attention, to separate whether the sink is used locally from whether it is structurally necessary at long range.

What I found

01
The query pathway matters. Query-side interventions transfer sink behaviour within a GQA group, identifying the query pathway as a causal contributor to within-group sink specialisation.

$Three panels plotting transfer fraction, BOS logit and sink score against alpha, the share of the donor query blended into the recipient head. Sink score rises from about 0.23 at alpha 0 to about 0.32 at alpha 1.$
Partial transplant Blending a sink head’s query into another head step by step (α from 0 to 1). The recipient’s sink score (c) climbs from about 0.23 to about 0.32 as more of the donor query is mixed in.
02
Sinks form early, then settle. Across depth, the sink emerges early and strengthens gradually, after which sink heads redistribute their attention comparatively little and their trajectories decelerate.

Concentration across depth Sink cells (red) already put about two-thirds of their attention on the first token by layer 3, rising to around 0.8–0.9 in later layers (top right). Matched control cells (blue) stay well below until around layer 20.

Redistribution between layers How much the attention pattern changes from one layer to the next (Jensen–Shannon divergence). Sink cells (red) mostly shift less than control cells (blue), and settle to around 0.05–0.13 in the last layers.
03
Whether it matters depends on the regime. Removing sink attention barely affects full-context next-token prediction, yet under windowed attention, retaining just one sink token is enough to restore performance.

Full context How far next-token predictions shift when sink attention is removed, measured as KL divergence. It stays small at every position, mostly below 0.1.

Windowed attention Once the window starts dropping early tokens (position 128), loss jumps from about 3 to about 10 if no sink is kept (blue). Keeping just one sink token (orange) stays close to the full-context baseline.

Together these suggest the attention sink is better understood as a head-specific, layer-evolving routing phenomenon whose causes and consequences depend on the architecture and the attention regime.

What I learned

The project began much broader: I set out to compare how attention handles Vietnamese and English tokenisation, but the near-identical sink behaviour in both redirected me. If the behaviour was the same, the real question was the mechanism building it. That taught me the habit this project is built on: not to stop at what a model does, but to look at its inside parts, variables and architecture. Reading into Qwen’s GQA this way revealed the natural experiment that let me turn a vague interest into a specific, testable claim about the query pathway.

Where it started Mean sink score per head on English (blue) and Vietnamese (red) text. The profiles are nearly identical, which is what turned the project from comparing languages to asking how the sink is built.

It also taught me to be suspicious of my own results. My query transplant first showed perfect transfer, but looking at how I had set up the experiment made me realise the result was forced by construction, so it could not establish causality. This pushed me to design the ripple and partial-transplant controls, to produce evidence that was not guaranteed by construction.

Two panels showing transfer fraction of exactly 1.0 for every metric and every intervened layer. — Too perfect My first transplant result: transfer of exactly 1.0 on every metric and at every layer. That was a sign the setup forced the answer, not evidence of a cause.

Two panels plotting transfer against distance downstream from the patched layer. Transfer is 1 at distance 0 and drops to around 0 at every later layer. — The ripple control Measuring how far a transplant’s effect travels. Transfer is 1 at the patched layer and falls to around 0 straight after, so the effect is not simply copied down the network.

The process taught me to form hypotheses, design experiments and interpret results, and it prepared me to investigate how local mechanisms give rise to wider model behaviour.