TestsBenchmarks · Coding
NL2Repo-Bench
Generate an entire working repository from a natural-language specification, scored by the repo's test suite.
Measured by independent testersIndependent Reported by the makerVendor-reported
| Thinking levelSetting | |||||||
|---|---|---|---|---|---|---|---|
| 1 | DeepSeek V4.1 Flash | DeepSeek | 64.0% | Maxreasoning_effort=100 (max) | Maker's own figureVendor-reportedDeepSeek ↗ | DeepSeek Harness (Minimal) | 10 Sep 2026 |
| 2 | DeepSeek V4 Pro (0813) | DeepSeek | 61.5% | Maxreasoning_effort=max | Maker's own figureVendor-reportedDeepSeek ↗ | DeepSeek Harness (minimal mode) | 13 Aug 2026 |
| 3 | GLM-5.3 | Z.ai (Zhipu) | 58.0% | Maxreasoning_effort=max | Maker's own figureVendor-reportedZ.ai ↗ | — | 14 Aug 2026 |
| 4 | DeepSeek V4 Flash Vision Exp | DeepSeek | 57.7% | Maxreasoning_effort=max | Maker's own figureVendor-reportedDeepSeek ↗ | DeepSeek Harness (minimal mode) | 21 Aug 2026 |
| 5 | GLM-5.3-Flash | Z.ai (Zhipu) | 56.3% | Maxreasoning_effort=max (default) | Maker's own figureVendor-reportedZ.ai ↗ | — | 26 Aug 2026 |
| 6 | Qwen3.8 Max (0803) | Qwen (Alibaba) | 55.9% | Defaultthinking (default reasoning_effort=xhigh; eval setting not stated) | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Claude Code | 3 Aug 2026 |
| 7 | DeepSeek V4 Flash (0731) | DeepSeek | 54.2% | Maxreasoning_effort=max | Maker's own figureVendor-reportedDeepSeek ↗ | DeepSeek Harness (minimal mode) | 31 Jul 2026 |
| 8 | GLM-5.2 | Z.ai (Zhipu) | 48.9% | Defaultthinking (effort not stated) | Maker's own figureVendor-reportedZ.ai ↗ | — | 16 Jun 2026 |
| 9 | Qwen3.8 Flash | Qwen (Alibaba) | 48.1% | Defaultthinking (setting not stated) | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Claude Code | 26 Aug 2026 |
| 10 | Qwen3.7 Max | Qwen (Alibaba) | 47.2% | Defaultthinking (vendor recommends 'Reasoning effort is set to xhigh' system prompt for reasoning tasks) | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Claude Code | 16 May 2026 |
| 11 | GLM-5.1 | Z.ai (Zhipu) | 42.7% | Defaultthinking | Maker's own figureVendor-reportedZ.ai ↗ | — | 7 Apr 2026 |
| 12 | Qwen3.8 27B | Qwen (Alibaba) | 42.3% | Defaultthinking (default reasoning_effort=xhigh; eval setting not stated) | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Claude Code | 14 Aug 2026 |
| 13 | MiniMax M3 | MiniMax | 42.1% | Defaultsetting not stated | Maker's own figureVendor-reportedMiniMax ↗ | Claude Code | 1 Jun 2026 |
| 14 | Qwen3.7 Plus | Qwen (Alibaba) | 41.1% | Defaultthinking | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | — | 21 May 2026 |
| 15 | MiniMax M2.7 | MiniMax | 39.8% | Defaultsetting not stated | Maker's own figureVendor-reportedMiniMax ↗ | — | 18 Mar 2026 |
| 16 | DeepSeek V4 Flash (Preview, 0423) | DeepSeek | 39.4% | Defaultsetting not stated for preview columns | Maker's own figureVendor-reportedDeepSeek ↗ | — | 13 Aug 2026 |
| 17 | DeepSeek V4 Pro (Preview, 0423) | DeepSeek | 38.5% | Defaultsetting not stated for preview columns | Maker's own figureVendor-reportedDeepSeek ↗ | — | 13 Aug 2026 |
| 18 | Qwen3.6 Plus | Qwen (Alibaba) | 37.9% | Defaultthinking | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Claude Code | 2 Apr 2026 |
| 19 | Qwen3.6 27B | Qwen (Alibaba) | 36.2% | Defaultthinking | Maker's own figureVendor-reportedQwen (Alibaba) ↗ | Claude Code | 22 Apr 2026 |