69. Mish Activation

Mish is a self-regularized smooth activation function introduced in 2019 and popularized by YOLOv4. It outperforms ReLU and Swish on many computer vision benchmarks.

Definition:

mish(x) = x * tanh(softplus(x))
         = x * tanh(ln(1 + exp(x)))

Key properties compared to ReLU:

• Smooth: infinitely differentiable, no kinks
• Non-monotonic: small negative lobe around x ≈ −0.3 to −3 allows mild negative outputs
• Unbounded above: no saturation for positive inputs
• Bounded below: approaches 0 as x → −∞

Signature: def mish(x)

• x: input array (any shape)
• Returns: same shape as x