Causal Attention Mask

Autoregressive models (GPT family) must not attend to future tokens. A causal mask zeroes out the contribution from positions j > i so that token i can only attend to positions 0..i inclusive. The standard recipe is to add a large negative number to those entries before the softmax (use -1e9 here for numerical equivalence with the reference solution).

Implement causal masked attention.

Signature: def causal_attention(Q, K, V)

Q, K, V: (N, d_k) — query, key, value matrices (same sequence)
Returns: (N, d_v) — attention output where position i only attends to positions 0..i

Math

Attention (Q, K, V) = softmax (\frac{Q K ^{⊤}}{d _{k}} + M) V

75. Causal Attention Mask

75. Causal Attention Mask