TestsBenchmarks · Coding
SWE Atlas - Test Writing
Quality of unit/integration tests an agent writes for real codebases, graded against rubrics and mutation checks.
Measured by independent testersIndependent Reported by the makerVendor-reported
| Thinking levelSetting | |||||||
|---|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 | Anthropic | 67.0% | Extra highFable-5.1 (Claude Code) xHigh* | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 2 | Claude Opus 5 | Anthropic | 62.2% | Extra highOpus 5 (Claude Code) xHigh | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 3 | Claude Fable 5 | Anthropic | 55.6% | Extra highFable-5 (Claude Code) xHigh* | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 4 | Gemini 3.8 Flash | Google (Gemini / DeepMind) | 53.7% | DefaultGemini 3.8 Flash (Mini-SWE-Agent) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 5 | GPT-6 Astra | OpenAI | 50.7% | Extra highGPT 6 Astra (Codex) xHigh | Independent testIndependentScale AI SEAL ↗ | Codex | 1 Oct 2026 |
| 6 | Claude Opus 4.8 | Anthropic | 49.6% | Extra highOpus 4.8 (Claude Code) xhigh | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 7 | GPT-5.6 Sol | OpenAI | 45.9% | Extra highGPT-5.6-Sol (Codex) xHigh* | Independent testIndependentScale AI SEAL ↗ | Codex | 1 Oct 2026 |
| 8 | GPT-5.4 | OpenAI | 44.4% | Extra highGPT-5.4 (Codex) xHigh | Independent testIndependentScale AI SEAL ↗ | Codex | 1 Oct 2026 |
| 9 | GPT-5.5 | OpenAI | 42.6% | Extra highGPT-5.5 (Codex) xHigh | Independent testIndependentScale AI SEAL ↗ | Codex | 1 Oct 2026 |
| 10 | GLM-5.2 | Z.ai (Zhipu) | 41.5% | DefaultGLM 5.2 (Mini-SWE-Agent) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 11 | Muse Spark 1.1 | Meta (Meta Superintelligence Labs, Muse) | 41.5% | Extra highMuse Spark 1.1 (Mini-SWE-Agent) xHigh | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 12 | GPT-5.3-Codex | OpenAI | 39.0% | Extra highGPT 5.3 (Codex) xHigh | Independent testIndependentScale AI SEAL ↗ | Codex | 1 Oct 2026 |
| 13 | Claude Opus 4.7 | Anthropic | 38.5% | DefaultOpus 4.7 (Claude Code) | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 14 | Claude Opus 4.6 | Anthropic | 36.7% | DefaultOpus-4.6 (Claude Code) | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 15 | Claude Sonnet 4.6 | Anthropic | 31.8% | DefaultSonnet-4.6 (Claude Code) | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 16 | Muse Spark | Meta (Meta Superintelligence Labs, Muse) | 31.1% | DefaultMuse Spark | Independent testIndependentScale AI SEAL ↗ | — | 1 Oct 2026 |
| 17 | MiniMax M3 | MiniMax | 30.8% | Defaultsetting not stated | Maker's own figureVendor-reportedMiniMax ↗ | Claude Code | 1 Jun 2026 |
| 18 | Gemini 3 Flash Preview | Google (Gemini / DeepMind) | 30.3% | DefaultGemini-3-Flash (Mini-SWE) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 19 | Gemini 3.1 Pro (Preview) | Google (Gemini / DeepMind) | 29.8% | DefaultGemini-3.1-Pro (Mini-SWE) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 20 | GLM 5 | Z.ai (Zhipu) | 28.7% | DefaultGlm-5 (Mini-SWE) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 21 | DeepSeek V4 Pro (Preview, 0423) | DeepSeek | 27.1% | DefaultDeepseek V4 Pro (Mini-SWE-Agent) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 22 | Kimi K2.5 | Moonshot AI (Kimi) | 25.8% | DefaultKimi-K2.5 (Mini-SWE) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 23 | MiniMax M2.5 | MiniMax | 18.6% | DefaultMinimax-M2.5 (Mini-SWE) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |