TestsBenchmarks · Coding
SWE-bench Multilingual
Share of 300 real GitHub issues across 9 programming languages a model fixes (bash-only mini-SWE-agent runs).
Measured by independent testersIndependent Reported by the makerVendor-reported
| Thinking levelSetting | |||||||
|---|---|---|---|---|---|---|---|
| 1 | Claude Opus 5.5 | Anthropic | 93.9% | Maxadaptive thinking, effort=max | Maker's own figureVendor-reportedAnthropic ↗ | — | 22 Sep 2026 |
| 2 | Claude Sonnet 5.5 | Anthropic | 90.3% | Maxadaptive thinking, effort=max | Maker's own figureVendor-reportedAnthropic ↗ | — | 28 Sep 2026 |
| 3 | Claude Opus 5 | Anthropic | 89.5% | Maxadaptive thinking, effort=max | Maker's own figureVendor-reportedAnthropic ↗ | — | 24 Jul 2026 |
| 4 | Claude Fable 5.1 | Anthropic | 89.1% | Maxadaptive thinking, effort=max | Maker's own figureVendor-reportedAnthropic ↗ | — | 22 Sep 2026 |
| 5 | Claude Fable 5 | Anthropic | 86.6% | Maxadaptive thinking, effort=max | Maker's own figureVendor-reportedAnthropic ↗ | — | 1 Sep 2026 |
| 6 | Claude Opus 4.8 | Anthropic | 84.4% | Maxadaptive thinking, effort=max | Maker's own figureVendor-reportedAnthropic ↗ | — | 24 Jul 2026 |
| 7 | Qwen3.8 Flash | Qwen (Alibaba) | 81.0% | Defaultthinking (setting not stated) | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | mini-SWE-agent | 26 Aug 2026 |
| 8 | Claude Opus 4.7 | Anthropic | 80.5% | Maxadaptive thinking, effort=max | Maker's own figureVendor-reportedAnthropic ↗ | — | 28 May 2026 |
| 9 | Claude Sonnet 5 | Anthropic | 78.3% | Maxadaptive thinking, effort=max | Maker's own figureVendor-reportedAnthropic ↗ | — | 28 Sep 2026 |
| 10 | Qwen3.7 Max | Qwen (Alibaba) | 78.3% | Defaultthinking (vendor recommends 'Reasoning effort is set to xhigh' system prompt for reasoning tasks) | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Internal scaffold (bash + file-edit) | 16 May 2026 |
| 11 | Kimi K2.6 | Moonshot AI (Kimi) | 76.7% | Defaultthinking | Maker's own figureVendor-reportedMoonshot AI ↗ | In-house SWE-agent-derived framework (bash/createfile/insert/view/strreplace/submit) | 20 Apr 2026 |
| 12 | MiniMax M2.7 | MiniMax | 76.5% | Defaultsetting not stated | Maker's own figureVendor-reportedMiniMax ↗ | — | 18 Mar 2026 |
| 13 | DeepSeek V4 Pro (Preview, 0423) | DeepSeek | 76.2% | MaxThink Max | Maker's own figureVendor-reportedDeepSeek ↗ | — | 24 Apr 2026 |
| 14 | Qwen3.7 Plus | Qwen (Alibaba) | 75.8% | Defaultthinking | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Internal scaffold (bash + file-edit) | 21 May 2026 |
| 15 | Qwen3.6 Plus | Qwen (Alibaba) | 73.8% | Defaultthinking | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Internal scaffold (bash + file-edit) | 2 Apr 2026 |
| 16 | Qwen3.8 27B | Qwen (Alibaba) | 73.8% | Defaultthinking (default reasoning_effort=xhigh; eval setting not stated) | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | mini-SWE-agent | 26 Aug 2026 |
| 17 | DeepSeek V4 Flash (Preview, 0423) | DeepSeek | 73.3% | MaxThink Max | Maker's own figureVendor-reportedDeepSeek ↗ | — | 24 Apr 2026 |
| 18 | Gemini 3 Flash Preview | Google (Gemini / DeepMind) | 72.7% | Default | Independent testIndependentSWE-bench ↗ | mini-SWE-agent 2.0.0a0 | 13 Feb 2026 |
| 19 | Claude Opus 4.6 | Anthropic | 72.0% | Default | Independent testIndependentSWE-bench ↗ | mini-SWE-agent 2.0.0a0 | 13 Feb 2026 |
| 20 | Qwen3.6 27B | Qwen (Alibaba) | 71.3% | Defaultthinking | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Internal scaffold (bash + file-edit) | 22 Apr 2026 |
| 21 | Claude Opus 4.5 | Anthropic | 70.7% | Default | Independent testIndependentSWE-bench ↗ | mini-SWE-agent 2.0.0a0 | 13 Feb 2026 |
| 22 | GLM 5 | Z.ai (Zhipu) | 69.7% | Default | Independent testIndependentSWE-bench ↗ | mini-SWE-agent 2.0.0a0 | 13 Feb 2026 |
| 23 | Gemini 3 Pro | Google (Gemini / DeepMind) | 68.7% | Default | Independent testIndependentSWE-bench ↗ | mini-SWE-agent 2.0.0a0 | 13 Feb 2026 |
| 24 | MiniMax M2.5 | MiniMax | 68.3% | Default | Independent testIndependentSWE-bench ↗ | mini-SWE-agent 2.0.0a0 | 16 Feb 2026 |
| 25 | Kimi K2.5 | Moonshot AI (Kimi) | 67.3% | Default | Independent testIndependentSWE-bench ↗ | mini-SWE-agent 2.0.0a0 | 13 Feb 2026 |
| 26 | Claude Sonnet 4.5 | Anthropic | 67.0% | Default | Independent testIndependentSWE-bench ↗ | mini-SWE-agent 2.0.0a0 | 13 Feb 2026 |
| 27 | GPT-5.2 | OpenAI | 66.7% | High | Independent testIndependentSWE-bench ↗ | mini-SWE-agent 2.0.0a0 | 13 Feb 2026 |
| 28 | GPT-5.2-Codex | OpenAI | 66.3% | Default | Independent testIndependentSWE-bench ↗ | mini-SWE-agent 2.0.0 | 20 Feb 2026 |
| 29 | Claude Haiku 4.5 | Anthropic | 64.7% | Default | Independent testIndependentSWE-bench ↗ | mini-SWE-agent 2.0.0a0 | 13 Feb 2026 |
| 30 | DeepSeek V3.2 | DeepSeek | 59.0% | Default | Independent testIndependentSWE-bench ↗ | mini-SWE-agent 2.0.0a0 | 13 Feb 2026 |
| 31 | GPT-5 Mini | OpenAI | 39.7% | Default | Independent testIndependentSWE-bench ↗ | mini-SWE-agent 2.0.0a0 | 13 Feb 2026 |