pass@k — TorchedUp

Given n total samples drawn from a model and c of them correct, compute the unbiased estimator of pass@k — the probability that at least one of k random samples is correct.

Signature: def pass_at_k(n: int, c: int, k: int) -> float

This is the standard estimator from the HumanEval / Codex paper: 1 minus the probability that a random size-k subset of the n samples contains no correct example. See the math reference below for the closed form. Implement it in a way that stays numerically stable for large n (binomial coefficients overflow if computed directly), and handle the degenerate case where there are fewer incorrect samples than k.

Math

pass@ k = 1 - \frac{( k n - c )}{( k n )} = 1 - i = 0 \prod k - 1 \frac{n - c - i}{n - i}

Asked at

Given n total samples drawn from a model and c of them correct, compute the unbiased estimator of pass@k — the probability that at least one of k random samples is correct.

Signature: def pass_at_k(n: int, c: int, k: int) -> float

Math

pass@ k = 1 - \frac{( k n - c )}{( k n )} = 1 - i = 0 \prod k - 1 \frac{n - c - i}{n - i}

Asked at

169. pass@k

169. pass@k