Given n total samples drawn from a model and c of them correct, compute the unbiased estimator of pass@k — the probability that at least one of k random samples is correct.
Signature: def pass_at_k(n: int, c: int, k: int) -> float
Formula (from the HumanEval / Codex paper):
n - c < k: return 1.0 (every choice of k must include a correct sample)1 - prod_{i=0}^{k-1} (n - c - i) / (n - i)Use the multiplicative form to stay numerically stable.
Math
Asked at
Test Results