TorchedUp
ProblemsPremium
TorchedUp
pass@kMedium
ProblemsPremium

pass@k (unbiased estimator)

Given n total samples drawn from a model and c of them correct, compute the unbiased estimator of pass@k — the probability that at least one of k random samples is correct.

Signature: def pass_at_k(n: int, c: int, k: int) -> float

Formula (from the HumanEval / Codex paper):

  • If n - c < k: return 1.0 (every choice of k must include a correct sample)
  • Otherwise: 1 - prod_{i=0}^{k-1} (n - c - i) / (n - i)

Use the multiplicative form to stay numerically stable.

Math

Asked at

Python (numpy)0/3 runs today

Test Results

○pass@1 = c/n
○k near n
○large k🔒 Premium
Advertisement