1. Run the bench

Install Harbor 0.23.0, pull the V3 task images, and run the complete cohort against your (agent, model) of choice:

uv tool install harbor==0.23.0

harbor run -d gnucleus-ai/cad-bench@v3 \
  -a <your-agent> \
  -m <your-model> \
  --n-concurrent 4 \
  --n-attempts 1 \
  --max-retries 0 \
  --upload --public

V3 uses FreeCAD 1.1.0, gnucleus-freecad-validator 0.6.0 with the v2 scorer, and Harbor task schema 1.3. Keep the verifier separate, restrict agent networking to the selected model provider, and limit agent work to task-required files and state under /app. Failed or unscored trials receive zero reward.

Use the model's maximum available effort and the 9,000-second time limit. Record the effective effort reported by the agent; if a requested setting is unsupported, contact us before submitting. Verify the dataset digest before starting:

sha256:8a786725d3648a76282fc5868df06da3d5e00d603f283573a53430803465d78a

Submit one complete 100-task job with exactly one attempt per task and retries disabled. Preserve every failed trial in the cohort. Only narrowly defined infrastructure failures may be replaced after review, with the original attempt retained. Model, tool, timeout, verifier, and output failures remain zero.

Historical results remain available on Harbor: V2 and V1.

2. Publish the Harbor job

If you did not use --upload --public when running the bench, upload the completed local job afterwards:

harbor upload jobs/<job-directory> --public

The job and every trial must be readable without contributor credentials. Keep the full trial configuration, generated CAD, verifier output, agent log, and ATIF trajectory.json. A complete legacy trajectory.jsonl is also accepted. A pinned Hugging Face mirror is optional archival redundancy; Harbor remains authoritative.

3. Open a submission PR

Open a PR in the submission repository with your public Harbor job URL. Identify the benchmark as V3 and include the exact agent, agent version, model, effective reasoning effort, dataset digest, mean reward, and error count. Maintainers verify the complete 100-task cohort and task digests, inspect trajectories, spot-check the score with gnucleus-freecad-validator, and audit cost data before displaying the result. The repository's existing V2 manifest instructions apply to the historical version; contact the maintainers about the V3 submission format before preparing a manifest.