# Harbor benchmark tooling for Goose A small command-line tool for running and comparing terminal-bench-style benchmarks against different agent harnesses, models, and goose builds. ## Current results Latest `cmd.py list` snapshot across the runs in `runs/`. All `*-full` runs cover the full `terminal-bench/terminal-bench-2` dataset (89 tasks). `pass/fail/err/tout` is the per-status breakdown. `compute` is the sum of per-trial durations (parallelism unrolled), not wall clock — it's a stable measure of how much agent time a run cost regardless of host concurrency. `turns` is the total number of agent turns across all trials (one per assistant message / harness step). ``` job_name model rate compute in out turns cost pass/fail/err/tout ----------------------------------------------------------------------------------------------------------------------------------- claude-sonnet46-full claude-sonnet-4-6 55.1% 20.2h 102.3M 1.2M 3k $42.83 49/23/1/16 goose-1.30-sonnet46-full claude-sonnet-4-6 50.6% 23.7h 2.4M - 3k - 45/24/2/18 goose-sonnet46-full-code-mode claude-sonnet-4-6 57.3% 22.0h 63.3M 1.1M 3k $206.43 51/20/2/16 nemotron-full nemotron-3-nano-30b-a3b 1.1% 21.8h 9.5M 2.2M 1k - 1/64/2/22 opencode-sonnet46-full claude-sonnet-4-6 52.8% 22.2h 111.5M 1.6M 3k $70.30 47/23/0/19 pi-sonnet46-full claude-sonnet-4-6 47.2% 24.4h 114.4M 1.8M 3k $74.82 42/25/1/21 sonnet46-dev-only claude-sonnet-4-6 48.3% 23.2h 70.6M 1.2M 3k $229.19 43/25/2/19 sonnet46-full claude-sonnet-4-6 50.6% 22.5h 62.4M - 3k - 45/21/3/20 sonnet46-sum_codem claude-sonnet-4-6 57.3% 21.9h 78.1M 1.4M 3k $254.53 51/23/2/13 sonnet46-summon-full claude-sonnet-4-6 55.1% 23.5h 67.2M 1.0M 3k $217.28 49/19/3/18 ``` Quick read: - `goose-sonnet46-full-code-mode` and `sonnet46-sum_codem` (both run codemode, the latter also enabling summon) lead at **57.3%**. - Stock goose (`sonnet46-full`, `developer,todo`) lands at **50.6%**, roughly on par with `opencode` (52.8%) and ahead of `pi` (47.2%) on the same model. Notably, `pi` also burned the most compute (24.4h) — slowest *and* lowest scoring of the sonnet runs. - `claude-sonnet46-full` at **55.1%** is harbor's vanilla `Goose` harness (curl-installed) — useful sanity check that our `GooseBinaryAgent` adapter isn't leaving points on the floor. - `nemotron-full` solves 1 task using roughly the same compute budget but only ~1k turns (vs 3k for sonnet runs) — the small model gives up or loses tool-call structure earlier, so it doesn't even reach the 100-turn cap on most trials. ## Setup Requires `uv`, Docker, and `rsync` on the host. `cmd.py` is a [PEP 723 inline-uv script](https://peps.python.org/pep-0723/), so `uv` installs its Python deps (just `harbor` and `PyYAML`) on first run. Secrets live in a `.env` file. `cmd.py` looks for one in the current working directory first, then in this script's directory. Only the keys for the provider you're using need to be set: ``` ANTHROPIC_API_KEY=sk-ant-... OPENROUTER_API_KEY=sk-or-... DATABRICKS_HOST=https://... DATABRICKS_TOKEN=... OPENAI_API_KEY=sk-... ``` alternatively, you can just export them in the session where you run the benchmark ## Running a goose benchmark The `run` subcommand builds a harbor config that uses our `GooseBinaryAgent` adapter — it uploads your local goose binary into each task container, generates a `config.yaml` from the template with the requested extensions flipped on, runs the recipe, and streams JSON output. ```bash # Pin a specific binary, default everything else ./evals/harbor/cmd.py run /path/to/goose --job-name my-run # Different model ./evals/harbor/cmd.py run /path/to/goose \ --model anthropic/claude-opus-4-5 --job-name opus-run # OpenRouter ./evals/harbor/cmd.py run /path/to/goose \ --model openrouter/nvidia/nemotron-3-nano-30b-a3b \ --job-name nemotron-smoke # Subset of tasks (note: harbor wants the qualified form) ./evals/harbor/cmd.py run /path/to/goose \ --tasks terminal-bench/fix-git,terminal-bench/extract-elf \ --job-name smoke # Toggle which extensions are enabled in config.yaml ./evals/harbor/cmd.py run /path/to/goose \ --extensions developer,todo,codemode --job-name codemode-run # Double the per-task timeout (useful for rerunning AgentTimeoutError trials) ./evals/harbor/cmd.py run /path/to/goose \ --timeout-multiplier 2.0 \ --tasks terminal-bench/oom,terminal-bench/compile-vim \ --job-name oom-retry-2x ``` Defaults: - dataset: `terminal-bench/terminal-bench-2` - model: `anthropic/claude-sonnet-4-6` - extensions: `developer,todo` - concurrency: 4 - max turns: 100 - trials: 1 - installs `libgomp1` in each container (disable with `--no-install-goose-runtime-deps`) Use `--dry-run` to print the generated harbor config without launching. ## Running a non-goose harness Stock harnesses that harbor ships with (opencode, pi, aider, claude-code, ...) don't need our adapter — they install themselves in the container and read secrets from env. Write a harbor YAML config directly and call `harbor run`: ```yaml # opencode-sonnet46-full.yaml job_name: opencode-sonnet46-full jobs_dir: /path/to/goose/evals/harbor/runs # so cmd.py picks it up n_attempts: 1 n_concurrent_trials: 4 environment: type: docker force_build: false delete: true env: - ANTHROPIC_API_KEY=${ANTHROPIC_API_KEY} agents: - import_path: harbor.agents.installed.opencode:OpenCode model_name: anthropic/claude-sonnet-4-6 datasets: - name: terminal-bench/terminal-bench-2 ``` ```bash export ANTHROPIC_API_KEY=... uv tool install harbor harbor run -c opencode-sonnet46-full.yaml ``` The output lands under `evals/harbor/runs/opencode-sonnet46-full/`, alongside goose runs. `cmd.py list / show / compare` treats them identically — they're all harbor `TrialResult` JSON under the hood. For pi specifically you can lift the existing config we used: ```yaml agents: - import_path: harbor.agents.installed.pi:Pi model_name: anthropic/claude-sonnet-4-6 kwargs: thinking: "off" ``` ## Inspecting results `cmd.py list` shows every run under `runs/` with one line per job: ```bash ./evals/harbor/cmd.py list ``` Drill into a specific run: ```bash ./evals/harbor/cmd.py show # all tasks ./evals/harbor/cmd.py show --status error # filter by outcome ./evals/harbor/cmd.py show --status timeout ``` Drill into a single task in a single run: ```bash ./evals/harbor/cmd.py task ./evals/harbor/cmd.py task --tail 50 # tail agent log ``` Compare two runs head-to-head: ```bash ./evals/harbor/cmd.py compare # summary ./evals/harbor/cmd.py compare -v # plus per-task diffs ``` Delete runs: ```bash ./evals/harbor/cmd.py rm [ ...] # confirms by default ./evals/harbor/cmd.py rm -y # skip the prompt ``` ## Syncing runs between machines If you run benchmarks on a remote box and want to inspect them locally: ```bash # Pull everything ./evals/harbor/cmd.py pull tbench@douwe.com:/home/tbench/work/goose # Just specific jobs ./evals/harbor/cmd.py pull tbench@douwe.com:/home/tbench/work/goose \ --jobs sonnet46-full pi-sonnet46-full # Mirror exactly (delete local runs that aren't on the remote) ./evals/harbor/cmd.py pull tbench@douwe.com:/home/tbench/work/goose --delete ``` The remote argument is `user@host:/path/to/goose` — `pull` appends `evals/harbor/runs/` to it and rsyncs into the local `runs/`. ## A typical comparison workflow ```bash # Run two configurations on the remote (in screen / mosh / tmux) ssh tbench@douwe.com cd /home/tbench/work/goose ./evals/harbor/cmd.py run ./target/release/goose --job-name baseline ./evals/harbor/cmd.py run ./target/release/goose \ --extensions developer,todo,codemode --job-name codemode # Pull results locally ./evals/harbor/cmd.py pull tbench@douwe.com:/home/tbench/work/goose \ --jobs baseline codemode # Diff ./evals/harbor/cmd.py compare baseline codemode -v ``` For deeper per-task understanding (why did A pass and B fail on this one task?), see the `compare_tasks` skill under `.agents/skills/`. Delegate to it with the two job names and a task name and it will read both trajectories, the task spec, and the verifier output, then explain the mechanism behind the divergence.