TorchedUp
LearnBetaProblemsSystem DesignSoonPremium
TorchedUp
LearnBetaProblemsSystem DesignSoonPremium
←

188. LayerNorm: One-pass vs Two-pass

Medium

A naive LayerNorm reads the input three times (mean pass, variance pass, normalization pass). A Welford / one-pass implementation fuses mean and variance into a single streaming pass, then normalizes — only two reads of x total.

Signature: def layernorm_pass_savings(N: int, D: int, dtype_bytes: int) -> list

For a (N, D) input:

  • Two-pass total reads of x: 3 * N * D * dtype_bytes
  • One-pass total reads of x: 2 * N * D * dtype_bytes

Return [twopass_reads_bytes, onepass_reads_bytes, savings_bytes] (all ints).

Math

Var(x)=E[x2]−(E[x])2(Welford fuses both into one streaming pass)

Asked at

NumPy

import numpy as np

 

def layernorm_pass_savings(...):

    pass

🔒

Premium problem

Free accounts include problems #1–20. Upgrade to unlock the editor, hidden test cases, and reference solutions for every problem.

Upgrade to PremiumBack to problems

Already premium?