TestsBenchmarks · Coding
SWE-Bench Pro (public, v1)
Share of 731 long-horizon, multi-file tasks from copyleft open-source repos an agent resolves without breaking existing tests.
Measured by independent testersIndependent Reported by the makerVendor-reported
| Thinking levelSetting | |||||||
|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 5.5 | Anthropic | 89.9% | Maxadaptive thinking, effort=max | Maker's own figureVendor-reportedAnthropic ↗ | — | 22 Sep 2026 |
| 2 | Claude Sonnet 5.5 | Anthropic | 81.3% | Maxadaptive thinking, effort=max | Maker's own figureVendor-reportedAnthropic ↗ | — | 28 Sep 2026 |
| 3 | Claude Fable 5.1 | Anthropic | 81.2% | Maxadaptive thinking, effort=max | Maker's own figureVendor-reportedAnthropic ↗ | — | 22 Sep 2026 |
| 4 | Claude Fable 5 | Anthropic | 80.0% | Maxadaptive thinking, effort=max | Maker's own figureVendor-reportedAnthropic ↗ | — | 1 Sep 2026 |
| 5 | Claude Opus 5 | Anthropic | 79.2% | Maxadaptive thinking, effort=max | Maker's own figureVendor-reportedAnthropic ↗ | — | 24 Jul 2026 |
| 6 | Claude Opus 4.8 | Anthropic | 69.2% | Maxadaptive thinking, effort=max | Maker's own figureVendor-reportedAnthropic ↗ | — | 24 Jul 2026 |
| 7 | Qwen3.8 Max (0803) | Qwen (Alibaba) | 67.7% | Defaultthinking (default reasoning_effort=xhigh; eval setting not stated) | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Claude Code | 3 Aug 2026 |
| 8 | Grok 4.5 | SpaceXAI (formerly xAI) | 64.7% | Defaulteffort not stated (API default high) | Maker's own figureVendor-reportedSpaceXAI ↗ | — | 16 Jul 2026 |
| 9 | GPT-5.6 Sol | OpenAI | 64.6% | DefaultGPT-5.6 launch table (effort not stated; text cites max for headline results) | Maker's own figureVendor-reportedOpenAI ↗ | — | 9 Jul 2026 |
| 10 | Claude Opus 4.7 | Anthropic | 64.3% | Maxadaptive thinking, effort=max | Maker's own figureVendor-reportedAnthropic ↗ | — | 28 May 2026 |
| 11 | GPT-5.6 Terra | OpenAI | 63.4% | DefaultGPT-5.6 launch table (effort not stated; text cites max for headline results) | Maker's own figureVendor-reportedOpenAI ↗ | — | 9 Jul 2026 |
| 12 | Claude Sonnet 5 | Anthropic | 63.2% | Maxadaptive thinking, effort=max | Maker's own figureVendor-reportedAnthropic ↗ | — | 28 Sep 2026 |
| 13 | GPT-5.6 Luna | OpenAI | 62.7% | DefaultGPT-5.6 launch table (effort not stated; text cites max for headline results) | Maker's own figureVendor-reportedOpenAI ↗ | — | 9 Jul 2026 |
| 14 | Qwen3.8 Flash | Qwen (Alibaba) | 62.5% | Defaultthinking (setting not stated) | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Claude Code | 26 Aug 2026 |
| 15 | GLM-5.2 | Z.ai (Zhipu) | 62.1% | Defaultthinking (effort not stated) | Maker's own figureVendor-reportedZ.ai ↗ | OpenHands | 16 Jun 2026 |
| 16 | Qwen3.8 27B | Qwen (Alibaba) | 61.7% | Defaultthinking (default reasoning_effort=xhigh; eval setting not stated) | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Claude Code | 14 Aug 2026 |
| 17 | Muse Spark 1.1 | Meta (Meta Superintelligence Labs, Muse) | 61.5% | DefaultMuse Spark 1.1* | Independent testIndependentScale AI SEAL ↗ | — | 1 Oct 2026 |
| 18 | Qwen3.7 Max | Qwen (Alibaba) | 60.6% | Defaultthinking (vendor recommends 'Reasoning effort is set to xhigh' system prompt for reasoning tasks) | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Internal scaffold (bash + file-edit) | 16 May 2026 |
| 19 | GPT-5.5 | OpenAI | 59.4% | DefaultGPT-5.6 launch table (effort not stated; text cites max for headline results) | Maker's own figureVendor-reportedOpenAI ↗ | — | 9 Jul 2026 |
| 20 | GPT-5.4 | OpenAI | 59.1% | Extra highgpt-5.4 (xHigh)* | Independent testIndependentScale AI SEAL ↗ | — | 1 Oct 2026 |
| 21 | MiniMax M3 | MiniMax | 59.0% | Defaultsetting not stated | Maker's own figureVendor-reportedMiniMax ↗ | Claude Code | 1 Jun 2026 |
| 22 | Gemini 3.6 Flash | Google (Gemini / DeepMind) | 58.7% | Defaultdefault settings (API default thinking_level=medium) | Maker's own figureVendor-reportedGoogle ↗ | internal Antigravity harness | 21 Jul 2026 |
| 23 | Kimi K2.6 | Moonshot AI (Kimi) | 58.6% | Defaultthinking | Maker's own figureVendor-reportedMoonshot AI ↗ | In-house SWE-agent-derived framework (bash/createfile/insert/view/strreplace/submit) | 20 Apr 2026 |
| 24 | GLM-5.1 | Z.ai (Zhipu) | 58.4% | Defaultthinking | Maker's own figureVendor-reportedZ.ai ↗ | — | 7 Apr 2026 |
| 25 | Claude Sonnet 4.6 | Anthropic | 58.1% | Maxadaptive thinking, effort=max | Maker's own figureVendor-reportedAnthropic ↗ | — | 30 Jun 2026 |
| 26 | Qwen3.7 Plus | Qwen (Alibaba) | 57.6% | Defaultthinking | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Internal scaffold (bash + file-edit) | 21 May 2026 |
| 27 | Qwen3.6 Plus | Qwen (Alibaba) | 56.6% | Defaultthinking | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Internal scaffold (bash + file-edit) | 2 Apr 2026 |
| 28 | MiniMax M2.7 | MiniMax | 56.2% | Defaultsetting not stated | Maker's own figureVendor-reportedMiniMax ↗ | — | 18 Mar 2026 |
| 29 | DeepSeek V4 Pro (Preview, 0423) | DeepSeek | 55.4% | MaxThink Max | Maker's own figureVendor-reportedDeepSeek ↗ | — | 24 Apr 2026 |
| 30 | Gemini 3.5 Flash | Google (Gemini / DeepMind) | 55.1% | Defaultdefault settings (API default thinking_level=medium) | Maker's own figureVendor-reportedGoogle ↗ | internal Antigravity harness | 19 May 2026 |
| 31 | Muse Spark | Meta (Meta Superintelligence Labs, Muse) | 55.0% | DefaultMuse Spark* | Independent testIndependentScale AI SEAL ↗ | — | 1 Oct 2026 |
| 32 | Gemini 3.5 Flash-Lite | Google (Gemini / DeepMind) | 54.2% | Highhigh thinking | Maker's own figureVendor-reportedGoogle ↗ | internal Antigravity harness | 21 Jul 2026 |
| 33 | Gemini 3.1 Pro (Preview) | Google (Gemini / DeepMind) | 54.2% | HighThinking (High) | Maker's own figureVendor-reportedGoogle ↗ | — | 19 Feb 2026 |
| 34 | Qwen3.6 27B | Qwen (Alibaba) | 53.5% | Defaultthinking | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Internal scaffold (bash + file-edit) | 22 Apr 2026 |
| 35 | DeepSeek V4 Flash (Preview, 0423) | DeepSeek | 52.6% | MaxThink Max | Maker's own figureVendor-reportedDeepSeek ↗ | — | 24 Apr 2026 |
| 36 | Claude Opus 4.6 | Anthropic | 51.9% | Defaultclaude-opus-4-6 (thinking)* | Independent testIndependentScale AI SEAL ↗ | — | 1 Oct 2026 |
| 37 | Muse Glimmer 30B | Meta (Meta Superintelligence Labs, Muse) | 51.2% | HighHigh reasoning | Maker's own figureVendor-reportedMeta ↗ | — | 10 Aug 2026 |
| 38 | Claude Opus 4.5 | Anthropic | 45.9% | Defaultclaude-opus-4-5-20251101 | Independent testIndependentScale AI SEAL ↗ | — | 1 Oct 2026 |
| 39 | Gemini 3 Pro | Google (Gemini / DeepMind) | 43.3% | Defaultgemini-3-pro-preview | Independent testIndependentScale AI SEAL ↗ | — | 1 Oct 2026 |
| 40 | GPT-5.2-Codex | OpenAI | 41.0% | Defaultgpt-5.2-codex | Independent testIndependentScale AI SEAL ↗ | — | 1 Oct 2026 |
| 41 | Gemini 3.1 Flash-Lite | Google (Gemini / DeepMind) | 38.3% | Highhigh thinking | Maker's own figureVendor-reportedGoogle ↗ | internal Antigravity harness | 21 Jul 2026 |
| 42 | MiniMax M2.1 | MiniMax | 36.8% | Defaultminimax-2.1 | Independent testIndependentScale AI SEAL ↗ | — | 1 Oct 2026 |
| 43 | Gemini 3 Flash Preview | Google (Gemini / DeepMind) | 34.6% | Defaultgemini-3-flash | Independent testIndependentScale AI SEAL ↗ | — | 1 Oct 2026 |
| 44 | GPT-5.2 | OpenAI | 29.9% | Defaultgpt-5.2 | Independent testIndependentScale AI SEAL ↗ | — | 1 Oct 2026 |
| 45 | gpt-oss-120b | OpenAI | 16.2% | Defaultgpt-oss-120b | Independent testIndependentScale AI SEAL ↗ | — | 1 Oct 2026 |