September 8, 2026
•
Major Benchmark Release
CAD Bench v2: A Corrected Benchmark for the New Frontier
Today we are releasing Parametric CAD Bench v2 and freezing CAD Bench v1. The new release keeps the 100-task parametric CAD format while updating the modeling runtime, correcting the evaluation harness, and starting a fresh leaderboard with the latest generation of frontier models and agents.
Claude Fable 5.1 leads with 84.81%, only 0.02 points ahead of GPT-6 Astra at 84.78%. That difference is well within their overlapping 95% confidence intervals, so we treat the two results as statistically equivalent on this 100-task sample. Grok 4.6 follows at 82.21%.
The v2 leaderboard
Score is mean continuous task reward. A perfect result earns 1.0, partial results receive partial credit, and failed or unscored trials count as zero. Every row below contains 100 trials and links to its public Harbor run.
| Rank | Model | Agent | Score (95% CI) | Perfect | Known cost | Run |
|---|---|---|---|---|---|---|
| 1 | Claude Fable 5.1 (max) | Claude Code | 84.81% ± 4.13 | 46/100 | $198.58 | Receipt |
| 2 | GPT-6 Astra (max) | Codex | 84.78% ± 4.17 | 45/100 | $135.62 | Receipt |
| 3 | Grok 4.6 (xhigh) | Grok Build | 82.21% ± 4.77 | 47/100 | $75.02 | Receipt |
| 4 | Claude Opus 5 (max) | Claude Code | 79.52% ± 5.69 | 49/100 | $102.02 | Receipt |
| 5 | Kimi K3 (max) | mini-swe-agent | 76.18% ± 6.39 | 46/100 | $42.98 | Receipt |
| 6 | Claude Sonnet 5 (max) | Claude Code | 70.52% ± 7.58 | 49/100 | $227.55 | Receipt |
| 7 | GPT-5.6-Sol (max) | Codex | 70.34% ± 7.01 | 43/100 | $73.67 | Receipt |
| 8 | GPT-5.6 Terra (max) | Codex | 68.66% ± 7.19 | 47/100 | $64.53 | Receipt |
| 9 | Muse Spark 1.3 (max) | mini-swe-agent | 65.46% ± 8.03 | 47/100 | $75.68 | Receipt |
| 10 | GLM-5.3 (max) | mini-swe-agent | 64.04% ± 8.61 | 51/100 | $28.15 | Receipt |
What changed in v2
FreeCAD moved from 0.21.2 to 1.1.0. The current runtime matches a modern, pinned CAD stack and gives agents substantially more time for long modeling loops.
The verifier is isolated from the agent. In v1, the grader package, structured specification, and reference geometry lived in the same task image behind filesystem permissions. v2 moves all held-out assets into a separate verifier image. The agent receives FreeCAD and the prompt; the verifier receives the answer artifacts only after the run.
Reward reporting now follows Harbor's contract exactly. The canonical reward JSON contains a single reward value. Geometry, spec, and diagnostic reasons are written to a separate sidecar, which prevents debugging fields from being interpreted as competing rewards. v2 also declares the expected answer artifacts under Harbor schema 1.2 so missing output is handled consistently.
The validator is updated from 0.1.0 to 0.4.0. Missing answers, invalid files, symlinks, and scorer crashes follow explicit zero-reward paths, while the generated script and FCStd file are retained for review.
A broader frontier snapshot
The first v2 release covers Claude Fable 5.1, GPT-6 Astra, Grok 4.6, Claude Opus 5, Kimi K3, Claude Sonnet 5, GPT-5.6-Sol, GPT-5.6 Terra, Muse Spark 1.3, and GLM-5.3 across Claude Code, Codex, Grok Build, and mini-swe-agent. Grok 4.6 reaches 82.21% for $75.02, while Kimi K3 reaches 76.18% for $42.98. GLM-5.3 is the least expensive completed row at $28.15 and records the board's highest perfect-task count, 51, despite one failed trial.
Costs cover the model usage reported by each run and should be read alongside agent choice, runtime, and token coverage. They are useful operational measurements, not a normalized model price comparison.
Thank you to our contributors
We are grateful to sponge-cake-allday, Albert Yu, Eric Spencer, Xavier Peich, and Qinchen Xu for contributing public tasks and benchmark results. Their work helped expand the benchmark and uncover issues addressed in V2. You can review the contributions in the CAD Bench submissions repository.
Why v1 is frozen
The task count is still 100, but the runtime, validator, isolation boundary, artifact contract, and failure handling changed. That makes a v2 score a measurement on a new benchmark version, not a continuation of the v1 series. We are preserving the complete v1 table and its pinned result receipts, while closing it to new submissions. All new results should target v2.