SWE-Bench Pro (private/commercial set)

Share of 276 tasks from private startup codebases an agent resolves; tests generalization to code models have never seen.

% solvedHigher is betterCounts toward:Weights: Coding ×0.9The test's websiteOfficial page ↗

The best 8 · 8 models tested

Top 8 · best setting per model · 8 models, 8 results (8 independent)

Measured by independent testersIndependent Reported by the makerVendor-reported

SWE-Bench Pro (private/commercial set) leaderboard
				Thinking levelSetting
1	Muse Spark 1.1	Meta (Meta Superintelligence Labs, Muse)	51.5%	DefaultMuse Spark 1.1*	Independent testIndependentScale AI SEAL ↗	—	1 Oct 2026
2	Claude Opus 4.6	Anthropic	47.1%	Defaultclaude-opus-4-6 (thinking)*	Independent testIndependentScale AI SEAL ↗	—	1 Oct 2026
3	Muse Spark	Meta (Meta Superintelligence Labs, Muse)	44.7%	DefaultMuse Spark*	Independent testIndependentScale AI SEAL ↗	—	1 Oct 2026
4	GPT-5.4	OpenAI	43.4%	Extra highgpt-5.4(xHigh)*	Independent testIndependentScale AI SEAL ↗	—	1 Oct 2026
5	Gemini 3.1 Pro (Preview)	Google (Gemini / DeepMind)	32.2%	Defaultgemini-3.1-pro (thinking)*	Independent testIndependentScale AI SEAL ↗	—	1 Oct 2026
6	GPT-5.2-Codex	OpenAI	27.7%	DefaultGPT 5.2 Codex	Independent testIndependentScale AI SEAL ↗	—	1 Oct 2026
7	GPT-5.2	OpenAI	23.8%	DefaultGPT 5.2	Independent testIndependentScale AI SEAL ↗	—	1 Oct 2026
8	Gemini 3 Pro	Google (Gemini / DeepMind)	17.9%	DefaultGemini 3 Pro	Independent testIndependentScale AI SEAL ↗	—	1 Oct 2026