memind/docs/agent-run-worker-rollout-runbook.md

Agent Run Worker Rollout Runbook

Date: 2026-07-02

This runbook controls the long-running Tool Gateway / external agent-run-worker rollout.

Production safety rules:

• Do not delete PG data.
• Do not delete or rewrite MindSpace user data. On current 103 production, MindSpace Service root is /Users/john/MindSpace; /Users/john/Project/Memind/MindSpace is legacy compatibility/storage context. Confirm against 103 runtime topology before any file operation.
• Keep H5 public traffic on https://m.tkmind.cn.
• Treat the 105 tunnel path as legacy rollback transport, not the primary H5 path.
• Keep default production state closed unless a rollout window is explicitly active.

Default Closed State

Expected production state:

cd /Users/john/Project/Memind
curl -fsS https://m.tkmind.cn/api/runtime/status
node scripts/check-agent-run-worker.mjs
node scripts/runtime-slo-report.mjs

Expected values:

toolRuntime.codeRunPolicy.enabled=false
toolRuntime.codeRunsEnabled=false
toolRuntime.queue.autoDispatch=true
toolRuntime.queue.toolGateway.enabled=false
check-agent-run-worker.ok=true
check-agent-run-worker.expected=disabled
check-agent-run-worker.running=false
runtime-slo-report.ok=true
runtime-slo-report.failures=[]

Admin DB Policy (Phase 1.5)

Code-run user/taskType/client toggles can be managed in memindadm instead of rebuilding H5 for every change.

Layer	Controls
memindadm (h5_agent_code_run_config)	codeRun.enabled, clientEnabled, allowlists, pageDataDev.autodetect, generalAutodetect, requireValidation
Portal env	MEMIND_TOOL_GATEWAY_ENABLED, worker queue topology, emergency override via MEMIND_CODE_RUN_POLICY_SOURCE=env
H5 runtime	/auth/status.agentCodeRun → client autodetect without VITE_* rebuild

Admin API (memind_adm :8082):

GET/PATCH /admin-api/agent-code-run/config
GET       /admin-api/agent-code-run/runtime

Ops UI: super-admin console → Code Run (/ops/admin/agent-code-run).

One-time env → DB migration (dev/staging):

node scripts/migrate-agent-code-run-config-from-env.mjs --dry-run
node scripts/migrate-agent-code-run-config-from-env.mjs --updated-by <admin-user-uuid>

Verify effective policy after save:

curl -fsS http://127.0.0.1:8081/auth/status -H "Cookie: h5_user=<token>" | jq '.agentCodeRun'
curl -fsS http://127.0.0.1:8081/api/runtime/status | jq '.toolRuntime.codeRunPolicy'

Page Data dev loop (local only):

MEMIND_TOOL_GATEWAY_ENABLED=1 npm run dev:page-data-aider-loop -- --dry-run

See also: help-code01.md, help-code02.md.

One-user Canary Profile

Use only for a controlled john-user canary.

Portal .env additions:

MEMIND_AGENT_RUN_AUTODISPATCH=0
MEMIND_AGENT_CODE_RUNS_ENABLED=1
MEMIND_AGENT_CODE_RUNS_USER_IDS=a6fb1e97-2b0f-447b-b138-4561d8e5c53e
MEMIND_AGENT_CODE_RUN_TASK_TYPES=small_patch
MEMIND_AGENT_CODE_RUNS_REQUIRE_VALIDATION=1
MEMIND_TOOL_GATEWAY_ENABLED=0
MEMIND_TOOL_GATEWAY_DRY_RUN=0

Worker installer environment:

MEMIND_AGENT_RUN_WORKER_START=1
MEMIND_AGENT_RUN_WORKER_POLL_MS=3000
MEMIND_AGENT_RUN_WORKER_BATCH_SIZE=1
MEMIND_AGENT_RUN_QUEUE_CONCURRENCY=1
MEMIND_TOOL_GATEWAY_ENABLED=1
MEMIND_TOOL_GATEWAY_DRY_RUN=0
MEMIND_AGENT_RUN_WORKDIR_OVERRIDE=/Users/john/Project/memind_canary/<rollout-id>
MEMIND_AGENT_RUN_WORKDIR_USER_ID=a6fb1e97-2b0f-447b-b138-4561d8e5c53e

All canary code writes must use /Users/john/Project/memind_canary/....

All-user Gray Profile

Current production rollout can use all H5 users by leaving the user and task-type allowlists empty. H5 default code-run now auto-attaches receipt validation, so required validation stays enabled.

Portal .env additions:

MEMIND_AGENT_RUN_AUTODISPATCH=0
MEMIND_AGENT_CODE_RUNS_ENABLED=1
MEMIND_AGENT_CODE_RUNS_USER_IDS=
MEMIND_AGENT_CODE_RUN_TASK_TYPES=
MEMIND_AGENT_CODE_RUNS_REQUIRE_VALIDATION=1
MEMIND_AGENT_RUN_QUEUE_CONCURRENCY=2
MEMIND_AGENT_RUN_WORKER_BATCH_SIZE=2
MEMIND_AGENT_RUN_HEARTBEAT_MS=30000
MEMIND_AGENT_RUN_WORKER_EXPECT_RUNNING=1
MEMIND_TOOL_GATEWAY_ENABLED=0
MEMIND_TOOL_GATEWAY_DRY_RUN=0
VITE_AGENT_CODE_RUNS_ENABLED=1
VITE_AGENT_CODE_RUNS_AUTODETECT=1
VITE_AGENT_CODE_RUNS_USER_IDS=

Worker installer environment:

MEMIND_AGENT_RUN_WORKER_START=1
MEMIND_AGENT_RUN_WORKER_POLL_MS=3000
MEMIND_AGENT_RUN_WORKER_BATCH_SIZE=2
MEMIND_AGENT_RUN_QUEUE_CONCURRENCY=2
MEMIND_AGENT_RUN_HEARTBEAT_MS=30000
MEMIND_TOOL_GATEWAY_ENABLED=1
MEMIND_TOOL_GATEWAY_DRY_RUN=0

Expected values:

toolRuntime.codeRunPolicy.enabled=true
toolRuntime.codeRunPolicy.userAllowlist=[]
toolRuntime.codeRunPolicy.taskTypeAllowlist=[]
toolRuntime.codeRunPolicy.requireValidation=true
toolRuntime.codeRunsEnabled=true
toolRuntime.queue.autoDispatch=false
toolRuntime.queue.maxConcurrentRuns=2
toolRuntime.queue.toolGateway.enabled=false
check-agent-run-worker.expected=running
check-agent-run-worker.running=true
check-agent-run-worker.processes includes --limit 2

Enable Procedure

1. Backup .env.

backup_dir="/Users/john/Project/memind_backups/$(date +%Y%m%d-%H%M)-agent-run-worker-rollout"
mkdir -p "$backup_dir"
cp /Users/john/Project/Memind/.env "$backup_dir/.env.before"

2. Append the Portal .env additions from the canary profile.

3. Restart Portal.

launchctl kickstart -k "gui/$(id -u)/cn.tkmind.memind-portal"

4. Verify Portal policy.

curl -fsS https://m.tkmind.cn/api/runtime/status | node -e '
let s="";process.stdin.on("data",d=>s+=d);process.stdin.on("end",()=>{
  const j=JSON.parse(s);
  console.log(JSON.stringify({
    ok:j.ok,
    codeRunPolicy:j.toolRuntime?.codeRunPolicy,
    autoDispatch:j.toolRuntime?.queue?.autoDispatch,
    toolGateway:j.toolRuntime?.queue?.toolGateway?.enabled
  }, null, 2));
})'

5. Create canary repo.

canary_dir="/Users/john/Project/memind_canary/agent-run-worker-rollout-$(date +%Y%m%d-%H%M%S)"
mkdir -p "$canary_dir"
git -C "$canary_dir" init
printf '# rollout canary\n' > "$canary_dir/README.md"
git -C "$canary_dir" add README.md
git -C "$canary_dir" commit -m 'init rollout canary'

6. Install and start worker with the worker installer environment.

MEMIND_AGENT_RUN_WORKER_START=1 \
MEMIND_AGENT_RUN_WORKDIR_OVERRIDE="$canary_dir" \
MEMIND_AGENT_RUN_WORKDIR_USER_ID="a6fb1e97-2b0f-447b-b138-4561d8e5c53e" \
MEMIND_TOOL_GATEWAY_ENABLED=1 \
MEMIND_TOOL_GATEWAY_DRY_RUN=0 \
MEMIND_AGENT_RUN_QUEUE_CONCURRENCY=1 \
bash /Users/john/Project/Memind/scripts/install-agent-run-worker-agent.sh

7. Verify worker running.

MEMIND_AGENT_RUN_WORKER_EXPECT_RUNNING=1 node /Users/john/Project/Memind/scripts/check-agent-run-worker.mjs

The worker LaunchAgent must have KeepAlive=true during all-user gray so a clean worker exit does not leave code-run queue processing stopped.

Check stale running recovery before and after worker changes:

node /Users/john/Project/Memind/scripts/agent-run-worker.mjs --recover-stale --stale-ms 900000 --limit 5

Check worker heartbeat status during all-user gray:

MEMIND_AGENT_RUN_WORKER_EXPECT_RUNNING=1 node /Users/john/Project/Memind/scripts/check-agent-run-worker.mjs

Expected queue fields:

heartbeatMs=30000
oldestRunningHeartbeatAgeMs=0 when no running rows
runningWithoutHeartbeatCount=0 during normal operation

Runtime SLO report should also expose the same heartbeat summary:

node /Users/john/Project/Memind/scripts/runtime-slo-report.mjs

Expected SLO fields:

runtime.toolQueueSlo.heartbeatMs=30000
runtime.toolQueueSlo.oldestRunningHeartbeatAgeMs=0 when no running rows
runtime.toolQueueSlo.runningWithoutHeartbeatCount=0 during normal operation
failures=[] unless heartbeat age exceeds MEMIND_AGENT_RUN_HEARTBEAT_STALE_MS

For the 24h all-user gray soak, keep the daily SLO report unchanged and install the separate hourly snapshot agent:

MEMIND_RUNTIME_SLO_SOAK_INTERVAL_SECONDS=3600 \
MEMIND_RUNTIME_SLO_SOAK_RETENTION_DAYS=7 \
bash /Users/john/Project/Memind/scripts/install-runtime-slo-soak-agent.sh

Expected soak state:

LaunchAgent label=cn.tkmind.memind-runtime-slo-soak
StartInterval=3600
reportDir=/Users/john/Project/Memind/reports/runtime-slo-soak
latest JSON/Markdown snapshot includes runtime.toolQueueSlo.heartbeatMs

Apply only when the listed running rows are known stale:

node /Users/john/Project/Memind/scripts/agent-run-worker.mjs --recover-stale --apply-recovery --stale-ms 900000 --limit 5

8. Submit only code runs with validation metadata.

Required user message metadata:

{
  "metadata": {
    "memindRun": {
      "validation": {
        "expectedFile": {
          "path": "RESULT.md",
          "contains": "EXPECTED_TOKEN"
        }
      }
    }
  }
}

Pause Procedure

Pause worker only:

launchctl bootout "gui/$(id -u)/cn.tkmind.memind-agent-run-worker" 2>/dev/null || true
launchctl disable "gui/$(id -u)/cn.tkmind.memind-agent-run-worker" 2>/dev/null || true
node /Users/john/Project/Memind/scripts/check-agent-run-worker.mjs

Portal remains in canary mode until .env is restored.

Disable Procedure

1. Stop and disable worker.

launchctl bootout "gui/$(id -u)/cn.tkmind.memind-agent-run-worker" 2>/dev/null || true
launchctl disable "gui/$(id -u)/cn.tkmind.memind-agent-run-worker" 2>/dev/null || true

2. Restore .env.

cp "$backup_dir/.env.before" /Users/john/Project/Memind/.env
launchctl kickstart -k "gui/$(id -u)/cn.tkmind.memind-portal"

3. Reinstall disabled worker plist.

MEMIND_AGENT_RUN_WORKER_START=0 bash /Users/john/Project/Memind/scripts/install-agent-run-worker-agent.sh

4. Verify closed state.

curl -fsS https://m.tkmind.cn/api/runtime/status
node /Users/john/Project/Memind/scripts/check-agent-run-worker.mjs
node /Users/john/Project/Memind/scripts/runtime-slo-report.mjs

Incident Checks

If code tasks appear stuck:

node /Users/john/Project/Memind/scripts/check-agent-run-worker.mjs
tail -200 /Users/john/Library/Logs/memind-agent-run-worker.log
curl -fsS https://m.tkmind.cn/api/runtime/status

If oldestPendingAgeMs grows while worker is running:

• Check executor keys and model mapping.
• Check MEMIND_TOOL_GATEWAY_ENABLED=1 in worker plist.
• Check MEMIND_AGENT_RUN_WORKDIR_OVERRIDE points to canary or intended workspace.
• Do not delete queued/failed rows; inspect events first.

Auto-pause Guard

Install the guard after enabling the external worker:

bash /Users/john/Project/Memind/scripts/install-agent-run-guard-agent.sh

Default guard thresholds:

recent failures >= 3 in 10 minutes
oldest queued/retryable age >= 5 minutes
queued + retryable count >= 10
oldest running age >= 15 minutes

When a threshold trips, agent-run-guard.mjs --apply:

backs up .env
sets MEMIND_AGENT_CODE_RUNS_ENABLED=0
sets MEMIND_AGENT_RUN_AUTODISPATCH=1
stops and disables cn.tkmind.memind-agent-run-worker
kickstarts cn.tkmind.memind-portal

Read-only check:

node /Users/john/Project/Memind/scripts/agent-run-guard.mjs --dry-run

If validation fails:

• Treat it as a product correctness failure, not transport failure.
• Validation failed is non-retryable by design.
• Inspect tool_gateway_result.stdoutTail, stderrTail, and tool_gateway_validation_failed.

Restore Proof

Final closed proof should include:

codeRunPolicy.enabled=false
codeRunsEnabled=false
queue.autoDispatch=true
queue.toolGateway.enabled=false
check-agent-run-worker.ok=true
check-agent-run-worker.running=false
runtime-slo-report.failures=[]