TorchedUp
ProblemsPremium
TorchedUp
Adam Optimizer StepMedium
ProblemsPremium

Adam Optimizer Step

Implement a single Adam optimizer parameter update.

Signature: def adam_step(theta: np.ndarray, grad: np.ndarray, m: np.ndarray, v: np.ndarray, t: int, lr: float = 0.01, beta1: float = 0.9, beta2: float = 0.999, eps: float = 1e-8) -> np.ndarray

Return the updated theta_new after one step.

  • Update biased first moment: m = beta1*m + (1-beta1)*grad
  • Update biased second moment: v = beta2*v + (1-beta2)*grad^2
  • Bias correction: m_hat = m/(1-beta1^t), v_hat = v/(1-beta2^t)
  • Parameter update: theta -= lr * m_hat / (sqrt(v_hat) + eps)

Math

Asked at

Python (numpy)0/3 runs today

Test Results

○first step
○multi-param first step
○second step with momentum🔒 Premium
Advertisement