Improve benchmarking (#9637)

Signed-off-by: Douwe Osinga <douwe@squareup.com>
Co-authored-by: Douwe Osinga <douwe@squareup.com>

evals/harbor/.agents/skills/compare_tasks/SKILL.md

@@ -40,12 +40,23 @@ jq '{task_name, trial_name}' "$TRIAL_A_DIR/result.json"
 
 ### 2. Headline facts
 
-Pull these fields from each trial's `result.json`. The actual shape (harbor
-0.8 `TrialResult`):
+The fastest path is to let `cmd.py task` do it for you — it already prints
+status, reward, duration, tokens, turns, cost, error class, and the tail of
+the verifier stdout:
+
+```bash
+./evals/harbor/cmd.py task "$RUN_A" "$TASK"
+./evals/harbor/cmd.py task "$RUN_B" "$TASK"
+```
+
+Only drop to raw `jq` against `result.json` if you need a field `cmd.py task`
+doesn't print. The actual shape (harbor 0.8 `TrialResult`):
 
 ```bash
 jq '{
-  reward: (.verifier_result.rewards.reward // null),
+  reward: (.verifier_result.rewards.reward
+           // (.verifier_result.rewards | to_entries | .[0].value)
+           // null),
   rewards_all: .verifier_result.rewards,
   duration_seconds: ((.finished_at | fromdateiso8601) - (.started_at | fromdateiso8601)),
   input_tokens: .agent_result.n_input_tokens,
@@ -57,6 +68,10 @@ jq '{
 }' "$TRIAL_A_DIR/result.json"
 ```
 
+The `reward` fallback mirrors `reporter.trial_reward`: if the verifier
+didn't use the conventional `reward` key, take the first value in the
+`rewards` map.
+
 Derive status from those:
 
 - `pass` if `reward >= 1.0`
@@ -72,31 +87,62 @@ timed out during teardown, or it timed out after writing the correct answer).
 If we got points, count them. See `reporter.trial_status` for the canonical
 rule.
 
-Several `agent_result` fields are commonly `null` for older `GooseBinaryAgent`
-runs (notably `n_cache_tokens`, `n_output_tokens`, `cost_usd`). Don't treat
-that as a failure — just omit those facts from the comparison if missing on
-either side. The reporter has fallbacks that read goose's `complete` event
-from `agent/goose.txt`; you don't normally need to replicate them here.
+Several `agent_result` fields can be `null` depending on the harness
+(notably `n_cache_tokens`, `n_output_tokens`, `cost_usd` on some goose
+runs). Don't treat that as a failure — just omit those facts from the
+comparison if missing on either side. `cmd.py task` already applies
+harbor's fallbacks (reading goose's `complete` event from `agent/goose.txt`
+when the structured field is null), so its numbers are the right ones to
+report.
 
 ### 3. Read the task spec
 
 The task definitions are NOT in the harbor Python package. They are plain
-text files on disk, in harbor's dataset cache. Do not run `find /` or
+text files on disk, in harbor's task cache. Do not run `find /` or
 `pip show harbor` — that is the wrong direction.
 
-Find the task directory (works on Linux and macOS):
+Harbor caches under `~/.cache/harbor/` on every platform (it uses
+`Path("~/.cache/harbor").expanduser()` unconditionally — there is no
+`~/Library/Caches/harbor` on macOS, despite what you might expect).
+
+The on-disk layout for package-backed tasks (the common case — everything
+in `terminal-bench/terminal-bench-2` lands here) is:
+
+```
+~/.cache/harbor/tasks/packages/<org>/<task>/<digest>/
+```
+
+Note: no dataset name in the path. Tasks are keyed by org + task name +
+content digest, not by which dataset pulled them. The `<digest>` segment
+changes when the task is republished, so discover the dir rather than
+hardcoding:
 
 ```bash
-TASK_DIR=$(
-  ls -d ~/.cache/harbor/datasets/terminal-bench__terminal-bench-2__*/tasks/"$TASK"/ 2>/dev/null \
-  || ls -d ~/Library/Caches/harbor/datasets/terminal-bench__terminal-bench-2__*/tasks/"$TASK"/ 2>/dev/null
-)
+TASK_DIR=$(ls -d ~/.cache/harbor/tasks/packages/terminal-bench/"$TASK"/*/ 2>/dev/null | head -1)
 echo "$TASK_DIR"
 ls "$TASK_DIR"
 ```
 
-If both lookups return empty, the dataset hasn't been downloaded yet — bail
-out and report that, rather than guessing.
+If that's empty, the task could be from a different org or a git source —
+broaden the search. `find` returns the parent (one level above the
+digest), so descend one more level. Guard against `$PARENT` being empty,
+otherwise the glob expands to `/*/` and matches the filesystem root:
+
+```bash
+PARENT=$(find ~/.cache/harbor/tasks -type d -name "$TASK" 2>/dev/null | head -1)
+if [ -n "$PARENT" ]; then
+  TASK_DIR=$(ls -d "$PARENT"/*/ 2>/dev/null | head -1)
+fi
+```
+
+If both lookups come up empty, the task hasn't been downloaded on this
+machine — bail out and report that, rather than guessing. (Runs sync via
+`cmd.py pull` but the task cache does not, so a machine that only inspects
+results may never have the spec locally.)
+
+`~/.cache/harbor/datasets/` exists too but holds dataset-level metadata,
+not the per-task `instruction.md` / `tests/` / `solution/` files — not
+what you want here.
 
 Inside, you care about three files:
 
@@ -115,11 +161,12 @@ Two sources, prefer the first when present:
 
 - `$TRIAL_DIR/agent/trajectory.json` — harbor's ATIF format, one entry per
   agent step. `jq '.steps[] | {step_id, source, message, tool_calls: [.tool_calls[]?.function_name]}'`
-  gives a compact view. Recent goose runs (after the populate_context_post_run
-  fix) have this; older `GooseBinaryAgent` runs may not.
+  gives a compact view. Most current runs have it; some older harness
+  versions may not.
 - `$TRIAL_DIR/agent/<harness>.txt` — raw stream-json or log. The filename
-  matches the harness: `goose.txt`, `pi.txt`, `opencode.txt`,
-  `claude-code.txt`. `ls "$TRIAL_DIR/agent/"` to find it.
+  matches the harness (commonly `goose.txt` or `pi.txt`; other harnesses
+  use their own name). Don't guess — run `ls "$TRIAL_DIR/agent/"` and use
+  whatever `.txt` file is there.
 
 Skim, don't quote in full. For each agent identify:
 
@@ -170,10 +217,11 @@ Output markdown with these sections in order:
 
 ## Tools you'll need
 
+- `./evals/harbor/cmd.py task <run> <task>` for the headline numbers
 - `ls -d` to discover the `<task>__<suffix>` trial directories
-- `jq` for `result.json`
+- `jq` for any `result.json` field `cmd.py task` doesn't print
 - file reads against `$TRIAL_DIR/agent/` and `$TRIAL_DIR/verifier/`
-- file reads against the dataset cache (`~/.cache/harbor/datasets/...`)
+- `find ~/.cache/harbor/tasks` to locate the task spec
 
 No Python imports, no `harbor` package required. Everything you need is on
 disk as JSON / text files.

evals/harbor/recipes/analyze_bench_failure.yaml

@@ -0,0 +1,188 @@
+version: 1.0.0
+title: analyze a single harbor benchmark failure
+description: compare one task across two runs, theorize why the target failed, propose what could change
+author:
+  contact: douwe@block.xyz
+
+parameters:
+  - key: target
+    input_type: string
+    requirement: required
+    description: "the run we want to improve (typically a goose run)"
+  - key: reference
+    input_type: string
+    requirement: required
+    description: "the run that succeeded on this task"
+  - key: task
+    input_type: string
+    requirement: required
+    description: "bare task name, e.g. extract-elf (not terminal-bench/extract-elf)"
+
+extensions:
+  - type: builtin
+    name: developer
+    display_name: Developer
+    timeout: 600
+    bundled: true
+    description: Core tool for file operations, shell commands, and code analysis
+
+instructions: analyze why goose (the target run) failed a task that the reference run passed, and suggest what might change in goose to fix it
+
+prompt: |
+  you are analyzing a single harbor benchmark task where the reference run
+  succeeded and the target run (typically goose) failed. the goal is to
+  form a theory about *why* target failed and suggest what we could change
+  in goose to fix it. this is analysis, not implementation — no code
+  changes, no worktrees.
+
+  target run (the one that failed):     {{ target }}
+  reference run (the one that passed):  {{ reference }}
+  task:                                 {{ task }}
+
+  this recipe assumes it is launched from the root of the goose repo
+  (the current working directory contains `evals/harbor/`). all paths
+  below are relative to that.
+
+  ## step 1: headline facts
+
+  cmd.py task prints status, reward, duration, tokens, turns, cost, error,
+  and a tail of the verifier output. start there for both runs:
+
+  ```
+  ./evals/harbor/cmd.py task {{ reference }} {{ task }}
+  ./evals/harbor/cmd.py task {{ target }} {{ task }}
+  ```
+
+  ## step 2: find the trial directories
+
+  harbor 0.8 names trial dirs `<task>__<random-suffix>`. discover them
+  from disk — don't guess the suffix:
+
+  ```
+  TARGET_DIR=$(ls -d evals/harbor/runs/{{ target }}/{{ task }}__*/ 2>/dev/null | head -1)
+  REF_DIR=$(ls -d evals/harbor/runs/{{ reference }}/{{ task }}__*/ 2>/dev/null | head -1)
+  echo "target: $TARGET_DIR"
+  echo "ref:    $REF_DIR"
+  ```
+
+  if either is empty the run didn't include this task — stop and report.
+
+  ## step 3: read the task spec
+
+  the task definition lives in harbor's task cache. package-backed tasks
+  (the common case, including all of terminal-bench-2) land under
+  `~/.cache/harbor/tasks/packages/<org>/<task>/<digest>/`. the digest is
+  per task version and changes when the task is republished, so discover
+  the directory rather than guessing:
+
+  ```
+  TASK_DIR=$(ls -d ~/.cache/harbor/tasks/packages/terminal-bench/{{ task }}/*/ 2>/dev/null | head -1)
+  echo "$TASK_DIR"
+  ls "$TASK_DIR"
+  ```
+
+  if that's empty, fall back to a broader search in case the task came
+  from a git source or a different org. note that `find` returns the
+  parent (one level above the digest), so descend one more level. guard
+  against `$PARENT` being empty — otherwise the glob expands to `/*/` and
+  matches the filesystem root:
+
+  ```
+  PARENT=$(find ~/.cache/harbor/tasks -type d -name "{{ task }}" 2>/dev/null | head -1)
+  if [ -n "$PARENT" ]; then
+    TASK_DIR=$(ls -d "$PARENT"/*/ 2>/dev/null | head -1)
+  fi
+  ```
+
+  if both come up empty the task isn't cached locally — say so and continue
+  with what you can learn from the trial dirs alone (the verifier stdout
+  often reveals what was being checked).
+
+  read these three when present:
+
+  - `instruction.md` — what the agent was asked to do
+  - `tests/test_outputs.py` or `run-tests.sh` — what the verifier checks
+  - `solution/solution.sh` — the reference correct answer
+
+  when describing a failure later, **quote the assertion that failed**
+  rather than paraphrasing — paraphrase is where wrong conclusions sneak in.
+
+  ## step 4: read each agent's trajectory
+
+  two sources per trial, prefer the first:
+
+  - `$TRIAL_DIR/agent/trajectory.json` — harbor's ATIF format, one entry
+    per agent step. compact view:
+    `jq '.steps[] | {step_id, source, message, tool_calls: [.tool_calls[]?.function_name]}' "$TRIAL_DIR/agent/trajectory.json"`
+  - `$TRIAL_DIR/agent/<harness>.txt` — raw log. filename varies by harness
+    (commonly `goose.txt` or `pi.txt`). don't guess; run
+    `ls "$TRIAL_DIR/agent/"` and use whatever .txt is there.
+
+  for each side identify:
+
+  - the approach the agent took
+  - the final artifacts it left in the container (files created / modified)
+  - for the target (the failure), the failure mode — pick one:
+    - misread the spec (wrong assumption about input/output)
+    - right approach, shallow bug (off-by-one, wrong encoding, wrong path)
+    - ran out of clock — but note whether it was making real progress or
+      thrashing. a thrashing timeout is really a logic failure.
+    - diverged into an unproductive thread (debugging a non-issue)
+    - the verifier expected something the spec didn't telegraph
+
+  ## step 5: read the verifier output
+
+  `$TRIAL_DIR/verifier/test-stdout.txt` is usually the most diagnostic
+  file — it shows exactly which assertion failed and what the agent's
+  output looked like at that point.
+
+  ```
+  tail -80 "$TARGET_DIR/verifier/test-stdout.txt"
+  ```
+
+  ## step 6: look at goose source for a theory
+
+  the target is (typically) goose. once you have a failure mode, dig into
+  the goose source (the current working directory) to see if there's
+  something there that could plausibly be improved. relevant places
+  depending on what you saw:
+
+  - `crates/goose/src/agents/` — agent loop, tool-call handling,
+    context management
+  - `crates/goose/src/providers/` — provider-specific quirks (prompt
+    shape, streaming, tool-call format)
+  - `crates/goose-mcp/src/developer/` — the developer extension, where
+    most shell/file tools live
+  - `crates/goose/src/prompts/` and any system-prompt strings — what
+    we're telling the model about how to behave
+  - `crates/goose-cli/src/` — cli-side behavior (less likely to matter
+    for bench)
+
+  use `rg` to search; don't grep the world. if the reference run used a
+  different harness (e.g. pi, opencode, claude-code), think about what
+  that harness does differently — sometimes it's just a prompt difference,
+  sometimes it's a tool-shape difference, sometimes it's a timeout or
+  retry policy.
+
+  ## step 7: write up the analysis
+
+  produce markdown with these sections:
+
+  - **task** — one-line restatement of what the task wanted
+  - **outcome** — reference vs target headline (status, reward, duration,
+    turns) and which assertion the target failed on (quote it)
+  - **what reference did** — 2–4 sentences on the winning approach
+  - **what target did** — 2–4 sentences on the losing approach, with the
+    failure mode named
+  - **theory** — why target failed in mechanism terms, not vibes. "the
+    developer extension's text_editor truncates files >2MB and the task
+    output was 3MB" beats "goose got confused".
+  - **what we might change in goose** — concrete, but open-ended. could be
+    a prompt tweak, a tool behavior change, a default config, a new
+    capability, or "this is a one-off task quirk and not worth chasing".
+    cite the source files you looked at. it's fine to list more than one
+    candidate, and fine to say "not sure, would want to look at more
+    failures with this shape first".
+
+  stop there. no code changes, no PRs, no issues filed. the user will
+  triage the suggestions across all the tabs once everything has run.

evals/harbor/recipes/compare_bench_run.yaml

@@ -0,0 +1,98 @@
+version: 1.0.0
+title: compare harbor benchmark runs
+description: find tasks where one run succeeded but another failed, and fan out per-task analysis
+author:
+  contact: douwe@block.xyz
+
+parameters:
+  - key: target
+    input_type: string
+    requirement: required
+    description: "the run we want to improve (typically a goose run)"
+  - key: reference
+    input_type: string
+    requirement: required
+    description: "the run to learn from (the one that did better)"
+
+extensions:
+  - type: builtin
+    name: developer
+    display_name: Developer
+    timeout: 600
+    bundled: true
+    description: Core tool for file operations, shell commands, and code analysis
+
+instructions: orchestrate per-task analysis of a benchmark regression
+
+prompt: |
+  you are comparing two harbor benchmark runs to find tasks where the
+  `reference` run passed but the `target` run did not, then fanning out
+  one agent per such task to analyze why.
+
+  target run (the one we want to improve): {{ target }}
+  reference run (the one to learn from):   {{ reference }}
+
+  this recipe assumes it is launched from the root of the goose repo
+  (i.e. the current working directory contains `evals/harbor/`). all paths
+  below are relative to that.
+
+  step 1: locate the runs.
+
+  confirm both run directories exist:
+
+  ```
+  ls evals/harbor/runs/{{ target }}/
+  ls evals/harbor/runs/{{ reference }}/
+  ```
+
+  if either is missing, stop and tell the user.
+
+  step 2: find the divergent tasks.
+
+  use cmd.py compare with -v to get the per-task breakdown:
+
+  ```
+  ./evals/harbor/cmd.py compare {{ reference }} {{ target }} -v
+  ```
+
+  note the argument order: we pass `reference` as A and `target` as B, so
+  the "Only A solved" section is exactly the list we want — tasks the
+  reference passed and the target did not.
+
+  do NOT filter out timeouts. a timeout often masks a real failure — the
+  agent kept going down a wrong path until the clock ran out. the per-task
+  analyzer will call out timeouts that are genuinely "right approach, ran
+  out of clock" versus ones that are really logic failures wearing a
+  timeout costume.
+
+  step 3: show the user the list and ask for confirmation.
+
+  print the list of tasks that will be analyzed, with the count, and ask
+  for explicit agreement before launching. mention that each task will get
+  its own iTerm tab running `analyze_bench_failure.yaml`.
+
+  step 4: on agreement, launch one tab per task.
+
+  for each task in the list, run (substituting `$PWD` and `<TASK>`):
+
+  ```
+  REPO=$PWD
+  osascript <<APPLESCRIPT
+  tell application "iTerm"
+      tell current window
+          create tab with default profile
+          tell current session to write text "cd $REPO && goose run --recipe evals/harbor/recipes/analyze_bench_failure.yaml --interactive --params=target={{ target }} --params=reference={{ reference }} --params=task=<TASK>"
+      end tell
+  end tell
+  APPLESCRIPT
+  ```
+
+  the `cd $REPO` matters: new iTerm tabs open in the user's home directory
+  by default, but the analyzer recipe expects to run from the repo root.
+
+  put a 1 second sleep between launches, otherwise keystrokes may go to
+  the wrong tab. use the bare task name (e.g. `extract-elf`), not the
+  qualified form (`terminal-bench/extract-elf`) — the analyzer recipe and
+  cmd.py both expect the bare name.
+
+  after launching, print a one-line summary of how many tabs you opened.