feat: add agent run heartbeat to slo reports
This commit is contained in:
@@ -30,6 +30,7 @@
|
||||
- P8.1 Queue Lease / Stuck Run Recovery: 已完成第一步,worker dispatch 前自动回收超时 running run,新增 `--recover-stale` dry-run/apply 运维入口,生产 synthetic stale run 验证通过。
|
||||
- P8.2 Portal DB/Auth Transient Error Hardening: 已完成第一步,session attach、`/auth/status` 和 API auth middleware 已捕获 DB/auth 瞬时错误,生产部署后 live health、auth/status、runtime/status、guard 和 SLO 验证通过。
|
||||
- P8.3 Worker Lease Heartbeat: 已完成第一步,running run 执行期间写入 `worker_heartbeat` event,runtime/status、worker check、guard 和 stale recovery 均优先使用 heartbeat age,生产 synthetic 与真实 worker canary 验证通过。
|
||||
- P8.4 Long-running Gray Soak and Heartbeat SLO: 已完成第一步,runtime SLO JSON/Markdown 报告加入 agent-run heartbeat 摘要和 failure 条件,生产只读与临时报表写入验证通过;继续保持并发 2 soak,不扩大灰度。
|
||||
- P5.15 Active Stream TTL Reconcile: 已按用户要求跳过,暂不做报表/定时 reconcile。
|
||||
- P5 Worker Pool 运维化: 已完成第一步,Redis Router 支持 worker drain。
|
||||
- P5.9 First-token Latency EWMA: 已完成,StreamController 会把首个 SSE chunk 延迟写入 Redis,SLO 报告已展示。
|
||||
@@ -3015,6 +3016,73 @@ runtime/status:
|
||||
- heartbeat data includes `pid=31341`, `attempt=1`, `heartbeatMs=30000`
|
||||
- `tool_gateway_validation` passed for receipt and `public/p83-heartbeat-real-20260702043642.html`
|
||||
|
||||
### 2026-07-02 P8.4 Long-running Gray Soak and Heartbeat SLO
|
||||
|
||||
目标:
|
||||
|
||||
- 不扩大并发、不做 UI,先把 P8.3 heartbeat 纳入每日 SLO 报告。
|
||||
- 让 23:55 定时日报和手动 `runtime-slo-report.mjs` 都能暴露 running heartbeat 风险。
|
||||
- 为后续 24h all-user gray soak 提供稳定观测字段。
|
||||
|
||||
改动:
|
||||
|
||||
- `scripts/runtime-slo-report.mjs`:
|
||||
- `runtime.toolQueueSlo` 新增 heartbeat 摘要:
|
||||
- `heartbeatMs`
|
||||
- `maxConcurrentRuns`
|
||||
- `statusCounts`
|
||||
- `oldestRunningStartedAt`
|
||||
- `oldestRunningAgeMs`
|
||||
- `oldestRunningHeartbeatAt`
|
||||
- `oldestRunningHeartbeatAgeMs`
|
||||
- `runningWithoutHeartbeatCount`
|
||||
- `latestRunningRun`
|
||||
- failure 条件新增:
|
||||
- `agent_run_heartbeat_stale_<age>`: running heartbeat age 超过阈值。
|
||||
- `agent_run_missing_heartbeat_<count>`: running run 长时间缺失 heartbeat。
|
||||
- Markdown 报告的 Tool Queue 段落展示:
|
||||
- `heartbeatMs`
|
||||
- `oldestRunningHeartbeatAgeMs`
|
||||
- `runningWithoutHeartbeatCount`
|
||||
- `.env.example`:
|
||||
- 记录 `MEMIND_AGENT_RUN_HEARTBEAT_STALE_MS=90000`。
|
||||
- `docs/agent-run-worker-rollout-runbook.md`:
|
||||
- 记录 heartbeat SLO 检查命令和期望字段。
|
||||
|
||||
测试:
|
||||
|
||||
- `node --check scripts/runtime-slo-report.mjs` 通过。
|
||||
- 源码目录只读 SLO:
|
||||
- `ok=true`
|
||||
- `failures=[]`
|
||||
- `toolQueueSlo.heartbeatMs=30000`
|
||||
- `toolQueueSlo.runningWithoutHeartbeatCount=0`
|
||||
|
||||
生产部署:
|
||||
|
||||
- 备份:
|
||||
- `/Users/john/Project/memind_backups/20260702-123947-p84-heartbeat-slo`
|
||||
- 已部署:
|
||||
- `scripts/runtime-slo-report.mjs`
|
||||
- 未重启 Portal、worker、goosed 或 H5。
|
||||
- 现有 SLO LaunchAgent 保持:
|
||||
- `cn.tkmind.memind-runtime-slo-report`
|
||||
- 每日 `23:55`
|
||||
- `--write-report --prune --retention-days 30`
|
||||
|
||||
生产验证:
|
||||
|
||||
- 生产只读 SLO:
|
||||
- `ok=true`
|
||||
- `failures=[]`
|
||||
- `toolQueueSlo.heartbeatMs=30000`
|
||||
- `toolQueueSlo.statusCounts={}`
|
||||
- `toolQueueSlo.oldestRunningHeartbeatAgeMs=0`
|
||||
- `toolQueueSlo.runningWithoutHeartbeatCount=0`
|
||||
- 临时 report-dir 写入验证:
|
||||
- JSON 和 Markdown 均生成成功。
|
||||
- 两种格式均包含 `heartbeatMs`、`oldestRunningHeartbeatAgeMs`、`runningWithoutHeartbeatCount`。
|
||||
|
||||
## 回滚策略
|
||||
|
||||
- P0: 修改前保留 `server.mjs` 备份;如启动失败,恢复备份并 `launchctl kickstart` Portal。
|
||||
|
||||
Reference in New Issue
Block a user