141. Overlap Compute and Comm

In a pipelined training step, gradient AllReduce can run concurrently with backward compute. Compute the per-step time with and without overlap.

Signature: def pipeline_step_time(compute_ms: float, comm_ms: float, overlap: bool) -> float

• With overlap: max(compute_ms, comm_ms) — whichever is the bottleneck.
• Without overlap: compute_ms + comm_ms — strictly sequential.

Example:

• compute=100ms, comm=40ms, overlap=True → 100ms
• compute=100ms, comm=40ms, overlap=False → 140ms