LoRA Backward — TorchedUp

The LoRA branch routes x through a down-projection A and up-projection B with a constant alpha/r scale (see the math reference). Compute the gradients of the loss w.r.t. A and B (the only trainable matrices — W is frozen).

Signature: def lora_backward(x: np.ndarray, A: np.ndarray, B: np.ndarray, dL_dy: np.ndarray, alpha: float, r: int) -> tuple

Shapes:

x: (batch, in)
A: (r, in)
B: (out, r)
dL_dy: (batch, out) — upstream gradient w.r.t. the LoRA output

Returns: (dA, dB) with shapes (r, in) and (out, r) (each gradient has the same shape as its parameter).

Math

\frac{\partial L}{\partial B} = \frac{α}{r} \frac{\partial L}{\partial y}^{⊤} h, \frac{\partial L}{\partial A} = \frac{α}{r} (\frac{\partial L}{\partial y} B)^{⊤} x

Asked at

Signature: def lora_backward(x: np.ndarray, A: np.ndarray, B: np.ndarray, dL_dy: np.ndarray, alpha: float, r: int) -> tuple

Shapes:

x: (batch, in)
A: (r, in)
B: (out, r)
dL_dy: (batch, out) — upstream gradient w.r.t. the LoRA output

Returns: (dA, dB) with shapes (r, in) and (out, r) (each gradient has the same shape as its parameter).

Math

\frac{\partial L}{\partial B} = \frac{α}{r} \frac{\partial L}{\partial y}^{⊤} h, \frac{\partial L}{\partial A} = \frac{α}{r} (\frac{\partial L}{\partial y} B)^{⊤} x

Asked at

150. LoRA Backward

150. LoRA Backward