feat: add worker heartbeat for agent runs
This commit is contained in:
Your Name
@@ -29,6 +29,7 @@
|
||||
- P6.19 Task-level Artifact Validation: 已完成第一步，H5 code-run 会在 receipt 之外追加可推断的 `public/*.html` 目标文件校验，页面编辑 code-run 会追加 page-edit task receipt，生产真实 canary 三项 expectedFiles 全部通过。
|
||||
- P8.1 Queue Lease / Stuck Run Recovery: 已完成第一步，worker dispatch 前自动回收超时 running run，新增 `--recover-stale` dry-run/apply 运维入口，生产 synthetic stale run 验证通过。
|
||||
- P8.2 Portal DB/Auth Transient Error Hardening: 已完成第一步，session attach、`/auth/status` 和 API auth middleware 已捕获 DB/auth 瞬时错误，生产部署后 live health、auth/status、runtime/status、guard 和 SLO 验证通过。
|
||||
- P8.3 Worker Lease Heartbeat: 已完成第一步，running run 执行期间写入 `worker_heartbeat` event，runtime/status、worker check、guard 和 stale recovery 均优先使用 heartbeat age，生产 synthetic 与真实 worker canary 验证通过。
|
||||
- P5.15 Active Stream TTL Reconcile: 已按用户要求跳过，暂不做报表/定时 reconcile。
|
||||
- P5 Worker Pool 运维化: 已完成第一步，Redis Router 支持 worker drain。
|
||||
- P5.9 First-token Latency EWMA: 已完成，StreamController 会把首个 SSE chunk 延迟写入 Redis，SLO 报告已展示。
|
||||
@@ -2929,6 +2930,91 @@ runtime/status:
|
||||
- `public/p619-task-validation-20260702042911.html`
|
||||
- `.memind/agent-runs/p619-task-validation-20260702042911-page-edit.json`
|
||||
|
||||
### 2026-07-02 P8.3 Worker Lease Heartbeat
|
||||
|
||||
目标:
|
||||
|
||||
- 将 running run 的 lease 从单纯 `started_at + timeout` 升级为 heartbeat-aware。
|
||||
- 避免长任务仍在执行时被 stale recovery 误杀。
|
||||
- 让 runtime/status、worker check 和 auto-pause guard 能看到 running heartbeat age。
|
||||
|
||||
改动:
|
||||
|
||||
- `agent-run-gateway.mjs`:
|
||||
- 新增 `MEMIND_AGENT_RUN_HEARTBEAT_MS`，默认 `30000`。
|
||||
- run 被 claim 为 `running` 后立即写入 `worker_heartbeat` event，后续按 heartbeat interval 定时写入。
|
||||
- run 进入 terminal 或失败后停止 heartbeat timer。
|
||||
- `recoverStaleRunningRuns()` 使用 `COALESCE(latest worker_heartbeat, started_at)` 判断 stale。
|
||||
- recovery UPDATE 也复核最新 heartbeat，避免 select/update 之间误杀刚刷新 heartbeat 的 run。
|
||||
- `getQueueStatus()` 增加:
|
||||
- `heartbeatMs`
|
||||
- `oldestRunningHeartbeatAt`
|
||||
- `oldestRunningHeartbeatAgeMs`
|
||||
- `runningWithoutHeartbeatCount`
|
||||
- `latestRunningRun`
|
||||
- `scripts/agent-run-guard.mjs`:
|
||||
- auto-pause 的 running age 判断改为 heartbeat age。
|
||||
- queue 摘要增加 heartbeat 字段。
|
||||
- `scripts/check-agent-run-worker.mjs`:
|
||||
- queue 摘要增加 heartbeat 字段。
|
||||
- `.env.example` 和 worker rollout runbook:
|
||||
- 记录 `MEMIND_AGENT_RUN_HEARTBEAT_MS`。
|
||||
- all-user gray 下记录 `MEMIND_AGENT_RUN_WORKER_EXPECT_RUNNING=1`。
|
||||
|
||||
测试:
|
||||
|
||||
- `node --check agent-run-gateway.mjs scripts/agent-run-worker.mjs scripts/agent-run-guard.mjs scripts/check-agent-run-worker.mjs` 通过。
|
||||
- `node --test agent-run-gateway.test.mjs agent-run-routes.test.mjs` 通过，31 tests pass。
|
||||
- 新增测试:
|
||||
- worker dispatch 会写 `worker_heartbeat`。
|
||||
- stale recovery 不回收有新 heartbeat 的旧 running run。
|
||||
- queue status 展示 heartbeat age 与 missing heartbeat count。
|
||||
|
||||
生产部署:
|
||||
|
||||
- 备份:
|
||||
- `/Users/john/Project/memind_backups/20260702-123510-p83-worker-heartbeat`
|
||||
- 已部署:
|
||||
- bundled `server.mjs`
|
||||
- bundled `scripts/agent-run-worker.mjs`
|
||||
- `scripts/agent-run-guard.mjs`
|
||||
- `scripts/check-agent-run-worker.mjs`
|
||||
- 已重启:
|
||||
- `cn.tkmind.memind-portal`
|
||||
- `cn.tkmind.memind-agent-run-worker`
|
||||
- 生产 `.env` 追加:
|
||||
- `MEMIND_AGENT_RUN_WORKER_EXPECT_RUNNING=1`
|
||||
|
||||
生产验证:
|
||||
|
||||
- `https://mm.tkmind.cn/api/status` 返回 `ok`。
|
||||
- `/api/runtime/status.toolRuntime.queue`:
|
||||
- `maxConcurrentRuns=2`
|
||||
- `heartbeatMs=30000`
|
||||
- `statusCounts={}`
|
||||
- `oldestRunningHeartbeatAgeMs=0`
|
||||
- `runningWithoutHeartbeatCount=0`
|
||||
- `check-agent-run-worker.mjs`:
|
||||
- `ok=true`
|
||||
- `expected=running`
|
||||
- `running=true`
|
||||
- guard dry-run:
|
||||
- `ok=true`
|
||||
- `shouldPause=false`
|
||||
- synthetic recovery canary:
|
||||
- no-heartbeat run `042276ae-ecac-4b7c-9c53-c02154ea85cd`
|
||||
- request `p83-no-heartbeat-20260702043604`
|
||||
- recovery `considered=1`, `recovered=1`
|
||||
- stale event includes `heartbeatAt=null` and `heartbeatAgeMs`
|
||||
- fresh-heartbeat synthetic run `97a2ac16-0234-483d-be4f-836ba95a91e7` was not recovered and then deleted to avoid queue residue.
|
||||
- real worker heartbeat canary:
|
||||
- run `ad2894a7-dc02-49c6-a859-da6af95cb143`
|
||||
- request `p83-heartbeat-real-20260702043642`
|
||||
- status `succeeded`
|
||||
- event chain includes `worker_heartbeat` before `tool_gateway_dispatch`
|
||||
- heartbeat data includes `pid=31341`, `attempt=1`, `heartbeatMs=30000`
|
||||
- `tool_gateway_validation` passed for receipt and `public/p83-heartbeat-real-20260702043642.html`
|
||||
|
||||
## 回滚策略
|
||||
|
||||
- P0: 修改前保留 `server.mjs` 备份；如启动失败，恢复备份并 `launchctl kickstart` Portal。
|
||||
|
||||
Reference in New Issue
Block a user