LoRA Forward — TorchedUp

Implement the LoRA (Low-Rank Adaptation) forward pass. Instead of training the full weight matrix W, LoRA freezes W and learns a low-rank update B @ A (where A and B are skinny matrices of rank r).

Signature: def lora_forward(x: np.ndarray, W: np.ndarray, A: np.ndarray, B: np.ndarray, alpha: float, r: int) -> np.ndarray

Shapes:

x: (batch, in)
W: (out, in) — frozen base weight
A: (r, in) — down-projection
B: (out, r) — up-projection

Returns: the frozen base output plus the low-rank update — shape (batch, out). The update routes x through A (down-projection to rank r) and B (up-projection back to out), scaled by alpha / r. See the math reference below.

Math

y = x W^{⊤} + \frac{α}{r} x A^{⊤} B^{⊤}

Asked at

Signature: def lora_forward(x: np.ndarray, W: np.ndarray, A: np.ndarray, B: np.ndarray, alpha: float, r: int) -> np.ndarray

Shapes:

x: (batch, in)
W: (out, in) — frozen base weight
A: (r, in) — down-projection
B: (out, r) — up-projection

Math

y = x W^{⊤} + \frac{α}{r} x A^{⊤} B^{⊤}

Asked at

149. LoRA Forward

149. LoRA Forward