TestsBenchmarks · Coding
SWE Atlas - Refactoring
Share of realistic large refactoring tasks an agent completes correctly in real repos.
Measured by independent testersIndependent Reported by the makerVendor-reported
| Thinking levelSetting | |||||||
|---|---|---|---|---|---|---|---|
| 1 | GPT-6 Astra | OpenAI | 59.0% | Extra highGPT 6 Astra (Codex) xHigh | Independent testIndependentScale AI SEAL ↗ | Codex | 1 Oct 2026 |
| 2 | Claude Fable 5.1 | Anthropic | 56.7% | Extra highFable-5.1 (Claude Code) xHigh | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 3 | Claude Fable 5 | Anthropic | 54.8% | Extra highFable-5 (Claude Code) xHigh | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 4 | Claude Opus 4.7 | Anthropic | 48.6% | DefaultOpus-4.7 (Claude Code) | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 5 | Claude Opus 4.8 | Anthropic | 46.7% | DefaultOpus 4.8 (Claude Code) | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 6 | GPT-5.5 | OpenAI | 44.8% | Extra highGPT-5.5 (Codex) xHigh | Independent testIndependentScale AI SEAL ↗ | Codex | 1 Oct 2026 |
| 7 | Gemini 3.8 Flash | Google (Gemini / DeepMind) | 44.8% | DefaultGemini 3.8 Flash (Mini-SWE-Agent) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 8 | GPT-5.4 | OpenAI | 44.3% | Extra highGPT 5.4 (Codex) xHigh | Independent testIndependentScale AI SEAL ↗ | Codex | 1 Oct 2026 |
| 9 | GLM-5.2 | Z.ai (Zhipu) | 42.4% | DefaultGLM 5.2 (Mini-SWE-Agent) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 10 | GPT-5.3-Codex | OpenAI | 42.4% | Extra highGPT 5.3 (Codex) xHigh | Independent testIndependentScale AI SEAL ↗ | Codex | 1 Oct 2026 |
| 11 | Claude Opus 4.6 | Anthropic | 35.6% | DefaultOpus-4.6 (Claude Code) | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 12 | Gemini 3.1 Pro (Preview) | Google (Gemini / DeepMind) | 33.8% | DefaultGemini-3.1-Pro (Gemini CLI) | Independent testIndependentScale AI SEAL ↗ | Gemini CLI | 1 Oct 2026 |
| 13 | Claude Sonnet 4.6 | Anthropic | 32.2% | DefaultSonnet-4.6 (Claude Code) | Independent testIndependentScale AI SEAL ↗ | Claude Code | 1 Oct 2026 |
| 14 | GLM 5 | Z.ai (Zhipu) | 24.2% | DefaultGlm-5 (Mini-SWE-Agent) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 15 | Kimi K2.5 | Moonshot AI (Kimi) | 20.9% | DefaultKimi-K2.5 (Mini-SWE-Agent) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 16 | MiniMax M2.5 | MiniMax | 19.5% | DefaultMinimax-M2.5 (Mini-SWE-Agent) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |
| 17 | Gemini 3 Flash Preview | Google (Gemini / DeepMind) | 10.0% | DefaultGemini-3-Flash (Mini-SWE-Agent) | Independent testIndependentScale AI SEAL ↗ | mini-SWE-agent | 1 Oct 2026 |