feat: add agent run guard rollout controls

2026-07-02 11:36:46 +08:00
parent 212e163006
commit 80b762053b
6 changed files with 3662 additions and 1 deletions
@@ -0,0 +1,306 @@
+# Agent Run Worker Rollout Runbook
+
+Date: 2026-07-02
+
+This runbook controls the long-running Tool Gateway / external `agent-run-worker` rollout.
+
+Production safety rules:
+
+- Do not delete PG data.
+- Do not delete or rewrite `/Users/john/Project/Memind/MindSpace` user data.
+- Keep H5 public traffic on `https://mm.tkmind.cn`.
+- Treat `m.tkmind.cn` / 105 tunnel as legacy rollback transport, not the primary H5 path.
+- Keep default production state closed unless a rollout window is explicitly active.
+
+## Default Closed State
+
+Expected production state:
+
+```bash
+cd /Users/john/Project/Memind
+curl -fsS https://mm.tkmind.cn/api/runtime/status
+node scripts/check-agent-run-worker.mjs
+node scripts/runtime-slo-report.mjs
+```
+
+Expected values:
+
+```text
+toolRuntime.codeRunPolicy.enabled=false
+toolRuntime.codeRunsEnabled=false
+toolRuntime.queue.autoDispatch=true
+toolRuntime.queue.toolGateway.enabled=false
+check-agent-run-worker.ok=true
+check-agent-run-worker.expected=disabled
+check-agent-run-worker.running=false
+runtime-slo-report.ok=true
+runtime-slo-report.failures=[]
+```
+
+## One-user Canary Profile
+
+Use only for a controlled john-user canary.
+
+Portal `.env` additions:
+
+```bash
+MEMIND_AGENT_RUN_AUTODISPATCH=0
+MEMIND_AGENT_CODE_RUNS_ENABLED=1
+MEMIND_AGENT_CODE_RUNS_USER_IDS=a6fb1e97-2b0f-447b-b138-4561d8e5c53e
+MEMIND_AGENT_CODE_RUN_TASK_TYPES=small_patch
+MEMIND_AGENT_CODE_RUNS_REQUIRE_VALIDATION=1
+MEMIND_TOOL_GATEWAY_ENABLED=0
+MEMIND_TOOL_GATEWAY_DRY_RUN=0
+```
+
+Worker installer environment:
+
+```bash
+MEMIND_AGENT_RUN_WORKER_START=1
+MEMIND_AGENT_RUN_WORKER_POLL_MS=3000
+MEMIND_AGENT_RUN_WORKER_BATCH_SIZE=1
+MEMIND_AGENT_RUN_QUEUE_CONCURRENCY=1
+MEMIND_TOOL_GATEWAY_ENABLED=1
+MEMIND_TOOL_GATEWAY_DRY_RUN=0
+MEMIND_AGENT_RUN_WORKDIR_OVERRIDE=/Users/john/Project/memind_canary/<rollout-id>
+MEMIND_AGENT_RUN_WORKDIR_USER_ID=a6fb1e97-2b0f-447b-b138-4561d8e5c53e
+```
+
+All canary code writes must use `/Users/john/Project/memind_canary/...`.
+
+## All-user Gray Profile
+
+Current production rollout can use all H5 users by leaving the user and task-type allowlists empty. Validation metadata is accepted when present, but not required for the H5 default code-run path.
+
+Portal `.env` additions:
+
+```bash
+MEMIND_AGENT_RUN_AUTODISPATCH=0
+MEMIND_AGENT_CODE_RUNS_ENABLED=1
+MEMIND_AGENT_CODE_RUNS_USER_IDS=
+MEMIND_AGENT_CODE_RUN_TASK_TYPES=
+MEMIND_AGENT_CODE_RUNS_REQUIRE_VALIDATION=0
+MEMIND_TOOL_GATEWAY_ENABLED=0
+MEMIND_TOOL_GATEWAY_DRY_RUN=0
+VITE_AGENT_CODE_RUNS_ENABLED=1
+VITE_AGENT_CODE_RUNS_AUTODETECT=1
+VITE_AGENT_CODE_RUNS_USER_IDS=
+```
+
+Worker installer environment:
+
+```bash
+MEMIND_AGENT_RUN_WORKER_START=1
+MEMIND_AGENT_RUN_WORKER_POLL_MS=3000
+MEMIND_AGENT_RUN_WORKER_BATCH_SIZE=1
+MEMIND_AGENT_RUN_QUEUE_CONCURRENCY=1
+MEMIND_TOOL_GATEWAY_ENABLED=1
+MEMIND_TOOL_GATEWAY_DRY_RUN=0
+```
+
+Expected values:
+
+```text
+toolRuntime.codeRunPolicy.enabled=true
+toolRuntime.codeRunPolicy.userAllowlist=[]
+toolRuntime.codeRunPolicy.taskTypeAllowlist=[]
+toolRuntime.codeRunPolicy.requireValidation=false
+toolRuntime.codeRunsEnabled=true
+toolRuntime.queue.autoDispatch=false
+toolRuntime.queue.toolGateway.enabled=false
+check-agent-run-worker.expected=running
+check-agent-run-worker.running=true
+```
+
+## Enable Procedure
+
+1. Backup `.env`.
+
+```bash
+backup_dir="/Users/john/Project/memind_backups/$(date +%Y%m%d-%H%M)-agent-run-worker-rollout"
+mkdir -p "$backup_dir"
+cp /Users/john/Project/Memind/.env "$backup_dir/.env.before"
+```
+
+2. Append the Portal `.env` additions from the canary profile.
+
+3. Restart Portal.
+
+```bash
+launchctl kickstart -k "gui/$(id -u)/cn.tkmind.memind-portal"
+```
+
+4. Verify Portal policy.
+
+```bash
+curl -fsS https://mm.tkmind.cn/api/runtime/status | node -e '
+let s="";process.stdin.on("data",d=>s+=d);process.stdin.on("end",()=>{
+  const j=JSON.parse(s);
+  console.log(JSON.stringify({
+    ok:j.ok,
+    codeRunPolicy:j.toolRuntime?.codeRunPolicy,
+    autoDispatch:j.toolRuntime?.queue?.autoDispatch,
+    toolGateway:j.toolRuntime?.queue?.toolGateway?.enabled
+  }, null, 2));
+})'
+```
+
+5. Create canary repo.
+
+```bash
+canary_dir="/Users/john/Project/memind_canary/agent-run-worker-rollout-$(date +%Y%m%d-%H%M%S)"
+mkdir -p "$canary_dir"
+git -C "$canary_dir" init
+printf '# rollout canary\n' > "$canary_dir/README.md"
+git -C "$canary_dir" add README.md
+git -C "$canary_dir" commit -m 'init rollout canary'
+```
+
+6. Install and start worker with the worker installer environment.
+
+```bash
+MEMIND_AGENT_RUN_WORKER_START=1 \
+MEMIND_AGENT_RUN_WORKDIR_OVERRIDE="$canary_dir" \
+MEMIND_AGENT_RUN_WORKDIR_USER_ID="a6fb1e97-2b0f-447b-b138-4561d8e5c53e" \
+MEMIND_TOOL_GATEWAY_ENABLED=1 \
+MEMIND_TOOL_GATEWAY_DRY_RUN=0 \
+MEMIND_AGENT_RUN_QUEUE_CONCURRENCY=1 \
+bash /Users/john/Project/Memind/scripts/install-agent-run-worker-agent.sh
+```
+
+7. Verify worker running.
+
+```bash
+MEMIND_AGENT_RUN_WORKER_EXPECT_RUNNING=1 node /Users/john/Project/Memind/scripts/check-agent-run-worker.mjs
+```
+
+8. Submit only code runs with validation metadata.
+
+Required user message metadata:
+
+```json
+{
+  "metadata": {
+    "memindRun": {
+      "validation": {
+        "expectedFile": {
+          "path": "RESULT.md",
+          "contains": "EXPECTED_TOKEN"
+        }
+      }
+    }
+  }
+}
+```
+
+## Pause Procedure
+
+Pause worker only:
+
+```bash
+launchctl bootout "gui/$(id -u)/cn.tkmind.memind-agent-run-worker" 2>/dev/null || true
+launchctl disable "gui/$(id -u)/cn.tkmind.memind-agent-run-worker" 2>/dev/null || true
+node /Users/john/Project/Memind/scripts/check-agent-run-worker.mjs
+```
+
+Portal remains in canary mode until `.env` is restored.
+
+## Disable Procedure
+
+1. Stop and disable worker.
+
+```bash
+launchctl bootout "gui/$(id -u)/cn.tkmind.memind-agent-run-worker" 2>/dev/null || true
+launchctl disable "gui/$(id -u)/cn.tkmind.memind-agent-run-worker" 2>/dev/null || true
+```
+
+2. Restore `.env`.
+
+```bash
+cp "$backup_dir/.env.before" /Users/john/Project/Memind/.env
+launchctl kickstart -k "gui/$(id -u)/cn.tkmind.memind-portal"
+```
+
+3. Reinstall disabled worker plist.
+
+```bash
+MEMIND_AGENT_RUN_WORKER_START=0 bash /Users/john/Project/Memind/scripts/install-agent-run-worker-agent.sh
+```
+
+4. Verify closed state.
+
+```bash
+curl -fsS https://mm.tkmind.cn/api/runtime/status
+node /Users/john/Project/Memind/scripts/check-agent-run-worker.mjs
+node /Users/john/Project/Memind/scripts/runtime-slo-report.mjs
+```
+
+## Incident Checks
+
+If code tasks appear stuck:
+
+```bash
+node /Users/john/Project/Memind/scripts/check-agent-run-worker.mjs
+tail -200 /Users/john/Library/Logs/memind-agent-run-worker.log
+curl -fsS https://mm.tkmind.cn/api/runtime/status
+```
+
+If `oldestPendingAgeMs` grows while worker is running:
+
+- Check executor keys and model mapping.
+- Check `MEMIND_TOOL_GATEWAY_ENABLED=1` in worker plist.
+- Check `MEMIND_AGENT_RUN_WORKDIR_OVERRIDE` points to canary or intended workspace.
+- Do not delete queued/failed rows; inspect events first.
+
+## Auto-pause Guard
+
+Install the guard after enabling the external worker:
+
+```bash
+bash /Users/john/Project/Memind/scripts/install-agent-run-guard-agent.sh
+```
+
+Default guard thresholds:
+
+```text
+recent failures >= 3 in 10 minutes
+oldest queued/retryable age >= 5 minutes
+queued + retryable count >= 10
+oldest running age >= 15 minutes
+```
+
+When a threshold trips, `agent-run-guard.mjs --apply`:
+
+```text
+backs up .env
+sets MEMIND_AGENT_CODE_RUNS_ENABLED=0
+sets MEMIND_AGENT_RUN_AUTODISPATCH=1
+stops and disables cn.tkmind.memind-agent-run-worker
+kickstarts cn.tkmind.memind-portal
+```
+
+Read-only check:
+
+```bash
+node /Users/john/Project/Memind/scripts/agent-run-guard.mjs --dry-run
+```
+
+If validation fails:
+
+- Treat it as a product correctness failure, not transport failure.
+- Validation failed is non-retryable by design.
+- Inspect `tool_gateway_result.stdoutTail`, `stderrTail`, and `tool_gateway_validation_failed`.
+
+## Restore Proof
+
+Final closed proof should include:
+
+```text
+codeRunPolicy.enabled=false
+codeRunsEnabled=false
+queue.autoDispatch=true
+queue.toolGateway.enabled=false
+check-agent-run-worker.ok=true
+check-agent-run-worker.running=false
+runtime-slo-report.failures=[]
+```