TestsBenchmarks · Coding
DeepSWE
Pass@1 on 113 from-scratch, long-horizon engineering tasks across 91 repos and 5 languages, with hand-written behavioral verifiers.
Measured by independent testersIndependent Reported by the makerVendor-reported
| Thinking levelSetting | |||||||
|---|---|---|---|---|---|---|---|
| 1 | Gemini 4 Argon | Google (Gemini / DeepMind) | 78.8% | DefaultGemini 4 Argon | Independent testIndependentArtificial Analysis ↗ | Antigravity CLI | 1 Oct 2026 |
| 2 | DeepSeek V4.1 Flash | DeepSeek | 74.2% | Maxreasoning_effort=100 (max) | Maker's own figureVendor-reportedDeepSeek ↗ | mini-SWE-agent | 10 Sep 2026 |
| 3 | GPT-6 Astra | OpenAI | 74.1% | Extra highgpt-6-astra_xhigh | Independent testIndependentDeepSWE (Datacurve) via Epoch AI ↗ | mini-SWE-agent | 1 Oct 2026 |
| 4 | Gemini 3.8 Flash | Google (Gemini / DeepMind) | 73.8% | Highgemini-3.8-flash_high | Independent testIndependentDeepSWE (Datacurve) via Epoch AI ↗ | mini-SWE-agent | 1 Oct 2026 |
| 5 | Claude Opus 5 | Anthropic | 73.6% | Maxclaude-opus-5_max | Independent testIndependentDeepSWE (Datacurve) via Epoch AI ↗ | mini-SWE-agent | 1 Oct 2026 |
| 6 | GPT-6.1 Sol | OpenAI | 73.2% | Extra highGPT-6.1 Sol (xhigh) | Independent testIndependentArtificial Analysis ↗ | Codex | 1 Oct 2026 |
| 7 | Muse Spark 1.3 | Meta (Meta Superintelligence Labs, Muse) | 73.2% | Extra highMuse Spark 1.3 (xhigh) | Independent testIndependentArtificial Analysis ↗ | Muse Code | 1 Oct 2026 |
| 8 | GPT-5.6 Sol | OpenAI | 72.7% | Maxgpt-5.6-sol_max | Independent testIndependentDeepSWE (Datacurve) via Epoch AI ↗ | mini-SWE-agent | 1 Oct 2026 |
| 9 | Grok 4.7 | SpaceXAI (formerly xAI) | 72.6% | Extra highGrok 4.7 (xhigh) | Independent testIndependentArtificial Analysis ↗ | Grok Build | 1 Oct 2026 |
| 10 | Claude Sonnet 5.5 | Anthropic | 72.0% | MaxSonnet 5.5 (max) | Independent testIndependentArtificial Analysis ↗ | Claude Code | 1 Oct 2026 |
| 11 | Claude Fable 5 | Anthropic | 69.9% | Extra highclaude-fable-5_xhigh | Independent testIndependentDeepSWE (Datacurve) via Epoch AI ↗ | mini-SWE-agent | 1 Oct 2026 |
| 12 | GPT-5.6 Terra | OpenAI | 69.6% | Maxgpt-5.6-terra_max | Independent testIndependentDeepSWE (Datacurve) via Epoch AI ↗ | mini-SWE-agent | 1 Oct 2026 |
| 13 | GPT-6 Sol | OpenAI | 69.0% | MaxGPT-6 Sol (max) | Independent testIndependentArtificial Analysis ↗ | Codex | 1 Oct 2026 |
| 14 | GLM-5.3 | Z.ai (Zhipu) | 69.0% | Maxglm-5.3_max | Independent testIndependentDeepSWE (Datacurve) via Epoch AI ↗ | mini-SWE-agent | 1 Oct 2026 |
| 15 | Kimi K3 | Moonshot AI (Kimi) | 68.5% | Maxkimi-k3_max | Independent testIndependentDeepSWE (Datacurve) via Epoch AI ↗ | mini-SWE-agent | 1 Oct 2026 |
| 16 | Claude Opus 5.5 | Anthropic | 68.4% | MaxOpus 5.5 (max) | Independent testIndependentArtificial Analysis ↗ | Claude Code | 1 Oct 2026 |
| 17 | Grok 4.6 | SpaceXAI (formerly xAI) | 67.5% | Mediumgrok-4.6_medium | Independent testIndependentDeepSWE (Datacurve) via Epoch AI ↗ | mini-SWE-agent | 1 Oct 2026 |
| 18 | GPT-5.6 Luna | OpenAI | 67.2% | Maxgpt-5.6-luna_max | Independent testIndependentDeepSWE (Datacurve) via Epoch AI ↗ | mini-SWE-agent | 1 Oct 2026 |
| 19 | GPT-5.5 | OpenAI | 67.0% | Extra highgpt-5.5_xhigh | Independent testIndependentDeepSWE (Datacurve) via Epoch AI ↗ | mini-SWE-agent | 1 Oct 2026 |
| 20 | Gemini 3.7 Flash | Google (Gemini / DeepMind) | 65.5% | Mediumgemini-3.7-flash_medium | Independent testIndependentDeepSWE (Datacurve) via Epoch AI ↗ | mini-SWE-agent | 1 Oct 2026 |
| 21 | Claude Fable 5.1 | Anthropic | 64.3% | MaxFable 5.1 (max) (with fallback) | Independent testIndependentArtificial Analysis ↗ | Claude Code | 1 Oct 2026 |
| 22 | GPT-6 Luna | OpenAI | 63.7% | MaxGPT-6 Luna (max) | Independent testIndependentArtificial Analysis ↗ | Codex | 1 Oct 2026 |
| 23 | GLM-5.3-Flash | Z.ai (Zhipu) | 63.4% | Maxglm-5.3-flash_max | Independent testIndependentDeepSWE (Datacurve) via Epoch AI ↗ | mini-SWE-agent | 1 Oct 2026 |
| 24 | DeepSeek V4 Flash Vision Exp | DeepSeek | 59.3% | Maxreasoning_effort=max | Maker's own figureVendor-reportedDeepSeek ↗ | DeepSeek Harness (minimal mode) | 21 Aug 2026 |
| 25 | Claude Opus 4.8 | Anthropic | 59.0% | Maxclaude-opus-4-8_max | Independent testIndependentDeepSWE (Datacurve) via Epoch AI ↗ | mini-SWE-agent | 1 Oct 2026 |
| 26 | Qwen3.8 Flash | Qwen (Alibaba) | 58.7% | Defaultthinking (setting not stated) | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | mini-SWE-agent | 26 Aug 2026 |
| 27 | Qwen3.8 Max (0803) | Qwen (Alibaba) | 57.5% | Extra highqwen3.8-max_xhigh | Independent testIndependentDeepSWE (Datacurve) via Epoch AI ↗ | mini-SWE-agent | 1 Oct 2026 |
| 28 | DeepSeek V4 Pro (0813) | DeepSeek | 57.2% | MaxDeepSeek V4 Pro 0813 (max) | Independent testIndependentArtificial Analysis ↗ | Codex | 1 Oct 2026 |
| 29 | Qwen3.8 Max (0803) | Qwen (Alibaba) | 56.6% | Defaultthinking (default reasoning_effort=xhigh; eval setting not stated) | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Claude Code | 3 Aug 2026 |
| 30 | Muse Spark 1.2 | Meta (Meta Superintelligence Labs, Muse) | 54.9% | Extra highmuse-spark-1.2_xhigh | Independent testIndependentDeepSWE (Datacurve) via Epoch AI ↗ | mini-SWE-agent | 1 Oct 2026 |
| 31 | DeepSeek V4 Flash (0731) | DeepSeek | 54.3% | MaxDeepSeek V4 Flash 0731 (max) | Independent testIndependentArtificial Analysis ↗ | Codex | 1 Oct 2026 |
| 32 | Grok 4.5 | SpaceXAI (formerly xAI) | 54.0% | High | Maker's own figureVendor-reportedSpaceXAI ↗ | — | 12 Aug 2026 |
| 33 | Muse Spark 1.1 | Meta (Meta Superintelligence Labs, Muse) | 53.3% | Defaulteffort not stated | Maker's own figureVendor-reportedMeta ↗ | — | 9 Jul 2026 |
| 34 | Qwen3.8 Max (0902) | Qwen (Alibaba) | 51.0% | DefaultQwen3.8 Max | Independent testIndependentArtificial Analysis ↗ | Claude Code | 1 Oct 2026 |
| 35 | Gemini 3.6 Flash | Google (Gemini / DeepMind) | 48.6% | Highhigh thinking | Maker's own figureVendor-reportedGoogle ↗ | mini-swe-agent | 13 Aug 2026 |
| 36 | GLM-5.2 | Z.ai (Zhipu) | 46.2% | Defaultthinking (effort not stated) | Maker's own figureVendor-reportedZ.ai ↗ | mini-swe-agent | 16 Jun 2026 |
| 37 | Qwen3.8 27B | Qwen (Alibaba) | 42.2% | Defaultthinking (default reasoning_effort=xhigh; eval setting not stated) | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Claude Code | 14 Aug 2026 |
| 38 | Gemini 3.5 Flash | Google (Gemini / DeepMind) | 37.0% | Mediummedium thinking | Maker's own figureVendor-reportedGoogle ↗ | mini-swe-agent | 21 Jul 2026 |
| 39 | Qwen3.7 Max | Qwen (Alibaba) | 21.6% | Defaultthinking (setting not stated) | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Claude Code / mini-SWE-agent (best) | 3 Aug 2026 |
| 40 | Qwen3.7 Plus | Qwen (Alibaba) | 14.2% | Defaultsetting not stated | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Claude Code | 14 Aug 2026 |
| 41 | Qwen3.6 27B | Qwen (Alibaba) | 13.3% | Defaultsetting not stated | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Claude Code | 14 Aug 2026 |
| 42 | DeepSeek V4 Pro (Preview, 0423) | DeepSeek | 12.8% | Defaultsetting not stated for preview columns | Maker's own figureVendor-reportedDeepSeek ↗ | — | 13 Aug 2026 |
| 43 | Gemini 3.1 Pro (Preview) | Google (Gemini / DeepMind) | 12.0% | Defaultthinking level not stated (API default high) | Maker's own figureVendor-reportedGoogle ↗ | mini-swe-agent | 21 Jul 2026 |
| 44 | Muse Spark | Meta (Meta Superintelligence Labs, Muse) | 10.0% | Defaulteffort not stated | Maker's own figureVendor-reportedMeta ↗ | — | 9 Jul 2026 |
| 45 | DeepSeek V4 Flash (Preview, 0423) | DeepSeek | 7.3% | Defaultsetting not stated for preview columns | Maker's own figureVendor-reportedDeepSeek ↗ | — | 13 Aug 2026 |