1. Run the bench

Pull the task images from Harbor Hub and run them against your (agent, model) of choice:

harbor run -d gnucleus-ai/cad-bench@v2 \
  -a <your-agent> \
  -m <your-model> \
  --n-attempts 1 \
  --max-retries 0 \
  --upload --public

CAD Bench v2 runs the agent in FreeCAD 1.1.0 and scores its declared answer artifacts in a separate verifier image. The held-out reference, structured spec, and validator never enter the agent environment. Failed or unscored trials receive zero reward.

Submit one complete 100-task job with exactly one attempt per task and retries disabled. Preserve every failed trial in the cohort; do not selectively rerun or replace failures.

CAD Bench v1 is frozen and closed to new submissions. Its final results remain available on Harbor.

2. Publish the Harbor job

If you did not use --upload --public when running the bench, upload the completed local job afterwards:

harbor upload jobs/<job-directory> --public

The job and every trial must be readable without contributor credentials. Keep the full trial configuration, generated CAD, verifier output, agent log, and ATIF trajectory.json. A complete legacy trajectory.jsonl is also accepted. A pinned Hugging Face mirror is optional archival redundancy; Harbor remains authoritative.

3. Open a submission PR

Add a manifest YAML under submissions/v2/ in the submission repo, starting from the v2 example manifest. The manifest points at the public Harbor job and declares the exact agent, agent version, model, reasoning effort, dataset digest, mean reward, and error count. Maintainers verify the complete 100-task cohort and task digests, inspect trajectories, spot-check the score with gnucleus-freecad-validator, and audit cost data. The leaderboard row is staged hidden and displayed only after the manifest is merged. See the full submission contract.