TestsBenchmarks · Coding
ProgramBench (avg test pass rate)
Average share of hidden behavioral tests passed when re-implementing 200 programs from scratch; partial-credit view of ProgramBench.
Measured by independent testersIndependent Reported by the makerVendor-reported
| Thinking levelSetting | |||||||
|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 5 | Anthropic | 74.7% | Extra high | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 2 | Claude Opus 4.8 | Anthropic | 70.9% | Extra high | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 3 | Muse Spark 1.3 | Meta (Meta Superintelligence Labs, Muse) | 70.8% | Max | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 4 | GPT-5.6 Sol | OpenAI | 69.9% | Extra high | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 5 | GPT-5.5 | OpenAI | 69.5% | Extra high | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 6 | GLM-5.2 | Z.ai (Zhipu) | 64.6% | Default | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 7 | Gemini 3.7 Flash | Google (Gemini / DeepMind) | 61.2% | Default | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 8 | Muse Spark 1.2 | Meta (Meta Superintelligence Labs, Muse) | 57.2% | Extra high | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 9 | Gemini 3.6 Flash | Google (Gemini / DeepMind) | 55.7% | Default | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 10 | Claude Opus 4.7 | Anthropic | 55.1% | Extra high | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 11 | Gemini 3.5 Flash | Google (Gemini / DeepMind) | 53.6% | Default | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 12 | Claude Opus 4.6 | Anthropic | 52.1% | Default | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 13 | Claude Sonnet 4.6 | Anthropic | 47.5% | Default | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 14 | Muse Spark 1.1 | Meta (Meta Superintelligence Labs, Muse) | 47.0% | Extra high | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 15 | GPT-5.4 | OpenAI | 37.7% | Default | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 16 | Gemini 3.1 Pro (Preview) | Google (Gemini / DeepMind) | 36.4% | Default | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 17 | Gemini 3 Flash Preview | Google (Gemini / DeepMind) | 31.8% | Default | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 18 | Claude Haiku 4.5 | Anthropic | 30.0% | Default | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 19 | GPT-5.4 Mini | OpenAI | 16.4% | Default | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |
| 20 | GPT-5 Mini | OpenAI | 16.0% | Default | Independent testIndependentProgramBench ↗ | mini-SWE-agent | 28 Sep 2026 |