TestsBenchmarks · Coding
SWE Atlas - Codebase QnA
Rubric-graded accuracy answering deep technical questions about large real codebases using an agent harness.
Measured by independent testersIndependent Reported by the makerVendor-reported
| Thinking levelSetting | |||||||
|---|---|---|---|---|---|---|---|
| 1 | Claude Sonnet 5.5 | Anthropic | 66.9% | MaxSonnet 5.5 (max) | Independent testIndependentArtificial Analysis ↗ | Claude Code | 1 Oct 2026 |
| 2 | Claude Opus 5.5 | Anthropic | 66.4% | MaxOpus 5.5 (max) | Independent testIndependentArtificial Analysis ↗ | Claude Code | 1 Oct 2026 |
| 3 | Kimi K3 | Moonshot AI (Kimi) | 66.1% | DefaultKimi K3 | Independent testIndependentArtificial Analysis ↗ | Kimi Code CLI | 1 Oct 2026 |
| 4 | Claude Fable 5.1 | Anthropic | 65.9% | Extra highClaude Fable 5.1 XHigh + SWE-2 Medium | Independent testIndependentArtificial Analysis ↗ | Devin Fusion CLI | 1 Oct 2026 |
| 5 | Claude Opus 5 | Anthropic | 63.2% | Extra highOpus 5 (Claude Code) xHigh | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 6 | Grok 4.7 | SpaceXAI (formerly xAI) | 62.9% | Extra highGrok 4.7 (xhigh) | Independent testIndependentArtificial Analysis ↗ | Grok Build | 1 Oct 2026 |
| 7 | Qwen3.8 Max (0902) | Qwen (Alibaba) | 62.1% | DefaultQwen3.8 Max | Independent testIndependentArtificial Analysis ↗ | Claude Code | 1 Oct 2026 |
| 8 | GPT-6 Astra | OpenAI | 61.8% | MaxGPT-6 Astra (max) | Independent testIndependentArtificial Analysis ↗ | Codex | 1 Oct 2026 |
| 9 | DeepSeek V4 Pro (0813) | DeepSeek | 61.8% | MaxDeepSeek V4 Pro 0813 (max) | Independent testIndependentArtificial Analysis ↗ | Codex | 1 Oct 2026 |
| 10 | GPT-6.1 Sol | OpenAI | 61.0% | Extra highGPT-6.1 Sol (xhigh) | Independent testIndependentArtificial Analysis ↗ | Codex | 1 Oct 2026 |
| 11 | Muse Spark 1.3 | Meta (Meta Superintelligence Labs, Muse) | 59.4% | MaxMuse Spark 1.3 (max) | Independent testIndependentArtificial Analysis ↗ | Muse Code | 1 Oct 2026 |
| 12 | GLM-5.3 | Z.ai (Zhipu) | 59.4% | DefaultGLM-5.3 | Independent testIndependentArtificial Analysis ↗ | Opencode | 1 Oct 2026 |
| 13 | Grok 4.6 | SpaceXAI (formerly xAI) | 58.3% | Extra highGrok 4.6 (xhigh) | Independent testIndependentArtificial Analysis ↗ | Grok Build | 1 Oct 2026 |
| 14 | GPT-6 Sol | OpenAI | 57.5% | MaxGPT-6 Sol (max) | Independent testIndependentArtificial Analysis ↗ | Codex | 1 Oct 2026 |
| 15 | Claude Opus 4.8 | Anthropic | 57.3% | Extra highOpus 4.8 (Claude Code) xhigh | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 16 | Gemini 4 Argon | Google (Gemini / DeepMind) | 56.5% | DefaultGemini 4 Argon | Independent testIndependentArtificial Analysis ↗ | Antigravity CLI | 1 Oct 2026 |
| 17 | GPT-5.6 Sol | OpenAI | 54.0% | MaxGPT-5.6 Sol (max) | Independent testIndependentArtificial Analysis ↗ | Codex | 1 Oct 2026 |
| 18 | DeepSeek V4 Flash (0731) | DeepSeek | 51.3% | MaxDeepSeek V4 Flash 0731 (max) | Independent testIndependentArtificial Analysis ↗ | Codex | 1 Oct 2026 |
| 19 | GPT-5.6 Luna | OpenAI | 48.7% | MaxGPT-5.6 Luna (max) | Independent testIndependentArtificial Analysis ↗ | Codex | 1 Oct 2026 |
| 20 | GLM-5.2 | Z.ai (Zhipu) | 48.1% | DefaultGLM 5.2 (Mini-SWE-Agent) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 21 | Gemini 3.8 Flash | Google (Gemini / DeepMind) | 47.0% | DefaultGemini 3.8 Flash (Mini-SWE-Agent) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 22 | Muse Spark 1.2 | Meta (Meta Superintelligence Labs, Muse) | 46.2% | Extra high | Maker's own figureVendor-reportedMeta ↗ | mini-swe-agent | 2 Sep 2026 |
| 23 | GPT-5.5 | OpenAI | 45.4% | Extra highGPT 5.5 (Codex) xHigh | Independent testIndependentScale AI SEAL ↗ | Codex | 1 Oct 2026 |
| 24 | GPT-6 Luna | OpenAI | 44.4% | MaxGPT-6 Luna (max) | Independent testIndependentArtificial Analysis ↗ | Codex | 1 Oct 2026 |
| 25 | Muse Spark 1.1 | Meta (Meta Superintelligence Labs, Muse) | 42.2% | Extra highMuse Spark 1.1 (Mini-SWE-Agent) xHigh | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 26 | GPT-5.4 | OpenAI | 40.8% | Extra highGPT 5.4 (Codex) xHigh | Independent testIndependentScale AI SEAL ↗ | Codex | 1 Oct 2026 |
| 27 | Claude Opus 4.7 | Anthropic | 40.3% | DefaultOpus 4.7 (Claude Code) | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 28 | Claude Fable 5 | Anthropic | 39.0% | Extra highFable-5 (Claude Code) xHigh* | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 29 | MiniMax M3 | MiniMax | 37.9% | Defaultsetting not stated | Maker's own figureVendor-reportedMiniMax ↗ | mini-SWE-agent | 1 Jun 2026 |
| 30 | Claude Opus 4.6 | Anthropic | 33.3% | DefaultOpus 4.6 (Claude Code) | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 31 | GPT-5.3-Codex | OpenAI | 32.6% | Extra highGPT 5.3 (Codex) xHigh | Independent testIndependentScale AI SEAL ↗ | Codex | 1 Oct 2026 |
| 32 | Claude Sonnet 4.6 | Anthropic | 31.2% | DefaultSonnet 4.6 (Claude Code) | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 33 | DeepSeek V4 Pro (Preview, 0423) | DeepSeek | 27.1% | DefaultDeepSeek V4 Pro (Mini-SWE-Agent) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 34 | Muse Spark | Meta (Meta Superintelligence Labs, Muse) | 24.2% | DefaultMuse Spark | Independent testIndependentScale AI SEAL ↗ | — | 1 Oct 2026 |
| 35 | GLM 5 | Z.ai (Zhipu) | 20.5% | DefaultGlm 5 (Mini-SWE-Agent) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 36 | Gemini 3.1 Pro (Preview) | Google (Gemini / DeepMind) | 13.5% | DefaultGemini 3.1 Pro (Mini-SWE-Agent) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 37 | Kimi K2.5 | Moonshot AI (Kimi) | 13.1% | DefaultKimi K2.5 (Mini-SWE-Agent) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 38 | MiniMax M2.7 | MiniMax | 11.3% | Defaultsetting not stated | Maker's own figureVendor-reportedMiniMax ↗ | mini-SWE-agent | 1 Jun 2026 |
| 39 | MiniMax M2.5 | MiniMax | 10.3% | DefaultMinimax M2.5 (Mini-SWE-Agent) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 40 | Gemini 3 Flash Preview | Google (Gemini / DeepMind) | 8.2% | DefaultGemini 3 Flash (Mini-SWE-Agent) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |