September 24, 2026
•
Leaderboard Update
Parametric CAD Bench V3: Claude Opus 5.5 Takes the Lead
Today we are adding two Claude Opus 5.5 results to the Parametric CAD Bench V3 leaderboard. Claude Code reaches 61.03%, while mini-swe-agent with the multimodal bridge reaches 57.96%.
The two new rows rank first and second on the current leaderboard. GPT-6 Astra follows at 56.87% and Claude Fable 5.1 at 56.75%. The confidence intervals of all four rows overlap, so this ordering should not be treated as a statistically decisive separation.
The updated V3 leaderboard
Overall score is the mean continuous reward across all 100 tasks. Failed and unscored trials count as zero. Scored excludes trials ending in an exception, while Perfect means a reward of exactly 1.0. The 95% confidence interval is calculated over all 100 per-task rewards. Every row links to its public Harbor run.
| Rank | Model | Agent | Effort | Create | Create + Edit | Image-to-CAD | Overall (95% CI) | Scored | Perfect | Cost (USD) | Run |
|---|---|---|---|---|---|---|---|---|---|---|---|
1 | Claude Opus 5.5 | Claude Code 2.1.280 | max | 50.87 | 39.69 | 84.66 | 61.03% ± 7.53 | 99/100 | 9/100 | $573.94 | View run |
2 | Claude Opus 5.5 | mini-swe-agent 2.4.6 | max | 42.79 | 41.38 | 81.77 | 57.96% ± 7.82 | 93/100 | 8/100 | $498.28 | View run |
3 | GPT-6 Astra | Codex 0.154.0 | max | 52.26 | 44.37 | 69.70 | 56.87% ± 6.85 | 100/100 | 5/100 | $317.81 | View run |
4 | Claude Fable 5.1 | Claude Code 2.1.270 | max | 55.48 | 36.75 | 72.71 | 56.75% ± 7.18 | 98/100 | 7/100 | $1,056.04 | View run |
5 | Claude Opus 5 | Claude Code 2.1.270 | max | 49.48 | 36.52 | 62.64 | 50.86% ± 7.45 | 99/100 | 6/100 | $1,012.21 | View run |
6 | Gemini 3.8 Flash | mini-swe-agent 2.4.6 | high | 47.94 | 31.22 | 31.10 | 36.19% ± 6.97 | 94/100 | 4/100 | $271.85 | View run |
7 | Grok 4.7 | Grok Build 1.0.30 | high | 45.57 | 34.63 | 26.91 | 34.82% ± 6.64 | 100/100 | 3/100 | $670.22 | View run |
8 | GPT-5.6 Sol | Codex 0.154.0 | max | 36.23 | 24.78 | 29.20 | 29.98% ± 6.17 | 100/100 | 1/100 | $291.99 | View run |
9 | Grok 4.6 | Grok Build 1.0.30 | high | 41.61 | 33.45 | 13.00 | 27.72% ± 6.30 | 100/100 | 2/100 | $390.94 | View run |
10 | Gemini 3.8 Flash | Antigravity 1.2.7 | high | 43.28 | 25.32 | 17.45 | 27.56% ± 6.72 | 100/100 | 4/100 | $196.74 | View run |
11 | Kimi K3 | mini-swe-agent 2.4.6 | max | 39.11 | 23.49 | 15.36 | 24.92% ± 6.40 | 96/100 | 1/100 | $314.55 | View run |
12 | Muse Spark 1.3 | mini-swe-agent 2.4.6 | max | 33.61 | 24.33 | 8.84 | 20.92% ± 5.81 | 99/100 | 1/100 | $357.03 | View run |
13 | GPT-5.6 Terra | Codex 0.154.0 | max | 23.32 | 23.23 | 8.18 | 17.24% ± 5.84 | 99/100 | 0/100 | $215.12 | View run |
Opus 5.5 improves substantially over Opus 5
Holding Claude Code constant, the overall score rises from 50.86% with Opus 5 to 61.03% with Opus 5.5, a paired improvement of 10.18 percentage points (paired 95% CI: 5.22 to 15.13). The largest change is image-to-CAD, which rises by 22.02 points, from 62.64% to 84.66%. Creation improves by 1.39 points and create-and-edit by 3.17 points.
The newer model is also less expensive in this run. Known API cost falls from $1,012.21 to $573.94, a 43.3% reduction, while token use falls from 946.0M to 552.4M, a 41.6% reduction.
The two agent results are close
With Claude Opus 5.5 held constant, Claude Code leads mini-swe-agent by 3.08 percentage points overall, but the paired 95% interval runs from -1.15 to 7.31 points. Claude Code wins 27 tasks, mini-swe-agent wins 28, and 45 tie. Claude Code is 8.07 points higher on creation and 2.90 points higher on image-to-CAD; mini-swe-agent is 1.68 points higher on create-and-edit.
Claude Code produces 99 scored trials versus 93 for mini-swe-agent. The mini-swe-agent run costs $75.66 less, at $498.28 rather than $573.94.
New image-to-CAD highs
Both Opus 5.5 rows achieve the highest image-to-CAD scores reported on the leaderboard so far. Claude Code reaches 84.66% and mini-swe-agent reaches 81.77%, compared with the previous high of 72.71% from Claude Fable 5.1. The result reinforces the importance of both model capability and the way an agent exposes engineering drawings to the model.
The audit counts every final exception as zero and includes superseded retry usage in cost and token totals. The Claude Code row has one final output-limit error; the mini-swe-agent row has seven final errors. Claude Code also invokes its Opus 5 fallback on two zero-scoring tasks after unsuccessful Opus 5.5 calls. Those fallback calls add no score benefit, and their $9.83 cost is included in the row total.