55. SiLU / Swish Activation

Easy

SiLU (Sigmoid Linear Unit), also called Swish, is silu(x) = x * sigmoid(x). It is used in LLaMA, EfficientNet, and many modern architectures.

Unlike ReLU, SiLU is smooth and non-monotonic — it has a small negative lobe for inputs around −1.3 that can help learning by allowing a soft gating signal to pass through rather than hard-zeroing it.

Signature: def silu(x: np.ndarray) -> np.ndarray

Math

SiLU (x) = x \cdot σ (x) = \frac{x}{1 + e ^{- x}}