diff --git a/docs/agent-run-worker-rollout-runbook.md b/docs/agent-run-worker-rollout-runbook.md index bfcf9a9..ed5a201 100644 --- a/docs/agent-run-worker-rollout-runbook.md +++ b/docs/agent-run-worker-rollout-runbook.md @@ -218,6 +218,23 @@ runtime.toolQueueSlo.runningWithoutHeartbeatCount=0 during normal operation failures=[] unless heartbeat age exceeds MEMIND_AGENT_RUN_HEARTBEAT_STALE_MS ``` +For the 24h all-user gray soak, keep the daily SLO report unchanged and install the separate hourly snapshot agent: + +```bash +MEMIND_RUNTIME_SLO_SOAK_INTERVAL_SECONDS=3600 \ +MEMIND_RUNTIME_SLO_SOAK_RETENTION_DAYS=7 \ +bash /Users/john/Project/Memind/scripts/install-runtime-slo-soak-agent.sh +``` + +Expected soak state: + +```text +LaunchAgent label=cn.tkmind.memind-runtime-slo-soak +StartInterval=3600 +reportDir=/Users/john/Project/Memind/reports/runtime-slo-soak +latest JSON/Markdown snapshot includes runtime.toolQueueSlo.heartbeatMs +``` + Apply only when the listed running rows are known stale: ```bash diff --git a/docs/architecture/memind-2-runtime-execution-assessment-20260702.md b/docs/architecture/memind-2-runtime-execution-assessment-20260702.md index c510ba5..0278317 100644 --- a/docs/architecture/memind-2-runtime-execution-assessment-20260702.md +++ b/docs/architecture/memind-2-runtime-execution-assessment-20260702.md @@ -19,12 +19,12 @@ - Stream Controller 已具备 SSE headers、abort propagation、backpressure pipeline。 - Redis Router 已启用,承担 worker runtime state。 - Goose Worker Pool 已从固定单点走向四 worker 可观测调度。 -- Aider/OpenHands 已从普通聊天默认能力中剥离,进入 code mode 和后端灰度门禁;P6.3 已把 code run 从 goosed session extension 外移到 `agent-run-v1` Tool Gateway 协议,P6.4 已完成 Aider 真实执行 canary,P6.5 已完成 OpenHands 真实执行 canary,P6.6 已完成 external worker 精确接管 code-run canary,P6.7 已加入 Tool Gateway 产物校验与输出审计,P6.8 已安装 external worker LaunchAgent,P6.9 已完成带 validation 的 external worker 灰度 canary,P6.10 已加入 external worker 只读观测脚本,P6.11 已加入放量策略门禁,P6.12 已开启全用户长期灰度并通过普通测试用户真实路径,P6.13 已安装自动暂停 guard,P6.15 已让 H5 code-run 自动补 receipt validation 并恢复后端 required validation,P6.17/P6.18 已把 external worker 并发 2 通过 canary 并固化为当前 all-user gray 策略,P6.19 已加入 task-level artifact validation,P8.1 已加入 stale running recovery,P8.2 已完成 Portal DB/Auth 瞬时错误兜底,P8.3 已加入 worker lease heartbeat,P8.4 已把 heartbeat 纳入 runtime SLO 报告。 +- Aider/OpenHands 已从普通聊天默认能力中剥离,进入 code mode 和后端灰度门禁;P6.3 已把 code run 从 goosed session extension 外移到 `agent-run-v1` Tool Gateway 协议,P6.4 已完成 Aider 真实执行 canary,P6.5 已完成 OpenHands 真实执行 canary,P6.6 已完成 external worker 精确接管 code-run canary,P6.7 已加入 Tool Gateway 产物校验与输出审计,P6.8 已安装 external worker LaunchAgent,P6.9 已完成带 validation 的 external worker 灰度 canary,P6.10 已加入 external worker 只读观测脚本,P6.11 已加入放量策略门禁,P6.12 已开启全用户长期灰度并通过普通测试用户真实路径,P6.13 已安装自动暂停 guard,P6.15 已让 H5 code-run 自动补 receipt validation 并恢复后端 required validation,P6.17/P6.18 已把 external worker 并发 2 通过 canary 并固化为当前 all-user gray 策略,P6.19 已加入 task-level artifact validation,P8.1 已加入 stale running recovery,P8.2 已完成 Portal DB/Auth 瞬时错误兜底,P8.3 已加入 worker lease heartbeat,P8.4 已把 heartbeat 纳入 runtime SLO 报告并安装小时级 soak 快照。 - PG 和 MindSpace 仍保持生产数据边界,SLO 报告只做统计读取;P6.3-P6.15 不新增 schema migration,不删除或修改既有用户数据。 整体执行评分: 9.95 / 10。 -可以支撑当前 H5 streaming 稳定性改造的基础目标。自动采样、worker sidecar heartbeat、SLO 只读快照、SLO 日报定时器、SLO 日报保留策略、first-token EWMA、first-token p50/p95 窗口趋势、Tool Gateway Queue v0、外部 worker 接管入口、真实 worker canary、后端 code-mode canary、code-run 用户级灰度 gate、H5 页面编辑 UI canary、P6.3 Tool Gateway 协议化、P6.4 Aider 真实 canary、P6.5 OpenHands 真实 canary、P6.6 external worker code-run canary、P6.7 Tool Gateway guardrails、P6.8 worker LaunchAgent、P6.9 validated external worker canary、P6.10 worker observability、P6.11 rollout policy gates、P6.12 all-user gray、P6.13 auto-pause guard、P6.15 H5 validation metadata、P6.17/P6.18 worker concurrency 2、P6.19 task-level artifact validation、P8.1 stale running recovery、P8.2 Portal DB/Auth transient hardening、P8.3 worker lease heartbeat 和 P8.4 heartbeat SLO 已经落地。P6.22 用户可见进度/失败说明按用户决策暂不开发,后续剩余差距转为 24h soak 观察和更细任务类型的校验策略扩展。 +可以支撑当前 H5 streaming 稳定性改造的基础目标。自动采样、worker sidecar heartbeat、SLO 只读快照、SLO 日报定时器、SLO 日报保留策略、first-token EWMA、first-token p50/p95 窗口趋势、Tool Gateway Queue v0、外部 worker 接管入口、真实 worker canary、后端 code-mode canary、code-run 用户级灰度 gate、H5 页面编辑 UI canary、P6.3 Tool Gateway 协议化、P6.4 Aider 真实 canary、P6.5 OpenHands 真实 canary、P6.6 external worker code-run canary、P6.7 Tool Gateway guardrails、P6.8 worker LaunchAgent、P6.9 validated external worker canary、P6.10 worker observability、P6.11 rollout policy gates、P6.12 all-user gray、P6.13 auto-pause guard、P6.15 H5 validation metadata、P6.17/P6.18 worker concurrency 2、P6.19 task-level artifact validation、P8.1 stale running recovery、P8.2 Portal DB/Auth transient hardening、P8.3 worker lease heartbeat 和 P8.4 heartbeat SLO/hourly soak 已经落地。P6.22 用户可见进度/失败说明按用户决策暂不开发,后续剩余差距转为 24h soak 结果复盘和更细任务类型的校验策略扩展。 ## 实测结果 @@ -1018,7 +1018,7 @@ Data boundary: ### P8.4 Long-running Gray Soak and Heartbeat SLO -结果: 通过,runtime SLO JSON/Markdown 已加入 agent-run heartbeat 摘要和 failure 条件,生产只读与临时报表写入验证均通过。 +结果: 通过,runtime SLO JSON/Markdown 已加入 agent-run heartbeat 摘要和 failure 条件,生产只读、临时报表写入与小时级 soak 快照安装验证均通过。 - `runtime.toolQueueSlo` 新增: - `heartbeatMs` @@ -1040,6 +1040,8 @@ Data boundary: - 生产部署: - backup `/Users/john/Project/memind_backups/20260702-123947-p84-heartbeat-slo` - deployed `scripts/runtime-slo-report.mjs` + - backup `/Users/john/Project/memind_backups/20260702-124520-p84-hourly-soak` + - deployed `scripts/install-runtime-slo-soak-agent.sh` - no service restart - 生产验证: - production read-only SLO `ok=true`, `failures=[]` @@ -1051,6 +1053,12 @@ Data boundary: - `cn.tkmind.memind-runtime-slo-report` - remains scheduled at `23:55` - writes with `--write-report --prune --retention-days 30` +- 小时级 soak: + - `cn.tkmind.memind-runtime-slo-soak` + - `StartInterval=3600` + - writes `reports/runtime-slo-soak` + - retention `7` days + - first snapshot `ok=true`, `failures=[]` ## 下一步执行建议 @@ -1063,4 +1071,5 @@ Data boundary: ### P8.4 Long-running Gray Soak and Heartbeat SLO - 第一阶段 SLO 字段已完成。 -- 下一步是 24h all-user gray soak 观察,不扩大并发。 +- 第二阶段安装独立小时级 soak 快照,不扩大并发。 +- 下一步是观察 24h 报告,再决定是否进入更细任务类型 validation 或并发策略调整。 diff --git a/docs/architecture/memind-2-streaming-agent-runtime-plan.md b/docs/architecture/memind-2-streaming-agent-runtime-plan.md index 126a211..404b179 100644 --- a/docs/architecture/memind-2-streaming-agent-runtime-plan.md +++ b/docs/architecture/memind-2-streaming-agent-runtime-plan.md @@ -30,7 +30,7 @@ - P8.1 Queue Lease / Stuck Run Recovery: 已完成第一步,worker dispatch 前自动回收超时 running run,新增 `--recover-stale` dry-run/apply 运维入口,生产 synthetic stale run 验证通过。 - P8.2 Portal DB/Auth Transient Error Hardening: 已完成第一步,session attach、`/auth/status` 和 API auth middleware 已捕获 DB/auth 瞬时错误,生产部署后 live health、auth/status、runtime/status、guard 和 SLO 验证通过。 - P8.3 Worker Lease Heartbeat: 已完成第一步,running run 执行期间写入 `worker_heartbeat` event,runtime/status、worker check、guard 和 stale recovery 均优先使用 heartbeat age,生产 synthetic 与真实 worker canary 验证通过。 -- P8.4 Long-running Gray Soak and Heartbeat SLO: 已完成第一步,runtime SLO JSON/Markdown 报告加入 agent-run heartbeat 摘要和 failure 条件,生产只读与临时报表写入验证通过;继续保持并发 2 soak,不扩大灰度。 +- P8.4 Long-running Gray Soak and Heartbeat SLO: 已完成第二步,runtime SLO JSON/Markdown 报告加入 agent-run heartbeat 摘要和 failure 条件,并安装独立小时级 soak 快照;继续保持并发 2 soak,不扩大灰度。 - P5.15 Active Stream TTL Reconcile: 已按用户要求跳过,暂不做报表/定时 reconcile。 - P5 Worker Pool 运维化: 已完成第一步,Redis Router 支持 worker drain。 - P5.9 First-token Latency EWMA: 已完成,StreamController 会把首个 SSE chunk 延迟写入 Redis,SLO 报告已展示。 @@ -3022,7 +3022,7 @@ runtime/status: - 不扩大并发、不做 UI,先把 P8.3 heartbeat 纳入每日 SLO 报告。 - 让 23:55 定时日报和手动 `runtime-slo-report.mjs` 都能暴露 running heartbeat 风险。 -- 为后续 24h all-user gray soak 提供稳定观测字段。 +- 为 24h all-user gray soak 提供稳定观测字段和小时级留存证据。 改动: @@ -3048,10 +3048,19 @@ runtime/status: - 记录 `MEMIND_AGENT_RUN_HEARTBEAT_STALE_MS=90000`。 - `docs/agent-run-worker-rollout-runbook.md`: - 记录 heartbeat SLO 检查命令和期望字段。 +- `scripts/install-runtime-slo-soak-agent.sh`: + - 新增独立 LaunchAgent installer。 + - 默认 label `cn.tkmind.memind-runtime-slo-soak`。 + - 默认 `StartInterval=3600`。 + - 默认写入 `reports/runtime-slo-soak`。 + - 默认保留 `7` 天。 +- `scripts/build-portal-runtime.mjs`: + - runtime artifact 会同步并 chmod soak installer。 测试: -- `node --check scripts/runtime-slo-report.mjs` 通过。 +- `bash -n scripts/install-runtime-slo-soak-agent.sh` 通过。 +- `node --check scripts/runtime-slo-report.mjs scripts/build-portal-runtime.mjs` 通过。 - 源码目录只读 SLO: - `ok=true` - `failures=[]` @@ -3062,13 +3071,20 @@ runtime/status: - 备份: - `/Users/john/Project/memind_backups/20260702-123947-p84-heartbeat-slo` + - `/Users/john/Project/memind_backups/20260702-124520-p84-hourly-soak` - 已部署: - `scripts/runtime-slo-report.mjs` + - `scripts/install-runtime-slo-soak-agent.sh` - 未重启 Portal、worker、goosed 或 H5。 - 现有 SLO LaunchAgent 保持: - `cn.tkmind.memind-runtime-slo-report` - 每日 `23:55` - `--write-report --prune --retention-days 30` +- 新增 soak LaunchAgent: + - `cn.tkmind.memind-runtime-slo-soak` + - `StartInterval=3600` + - `--write-report --prune --retention-days 7` + - report dir `/Users/john/Project/Memind/reports/runtime-slo-soak` 生产验证: @@ -3082,6 +3098,11 @@ runtime/status: - 临时 report-dir 写入验证: - JSON 和 Markdown 均生成成功。 - 两种格式均包含 `heartbeatMs`、`oldestRunningHeartbeatAgeMs`、`runningWithoutHeartbeatCount`。 +- soak 快照验证: + - LaunchAgent 已加载。 + - 首次 kickstart 快照已写入 JSON/Markdown。 + - 最新快照 `ok=true`, `failures=[]`。 + - JSON/Markdown 均包含 heartbeat summary。 ## 回滚策略 diff --git a/scripts/build-portal-runtime.mjs b/scripts/build-portal-runtime.mjs index 13ef6f9..8581753 100755 --- a/scripts/build-portal-runtime.mjs +++ b/scripts/build-portal-runtime.mjs @@ -319,6 +319,10 @@ async function writeMetadata() { path.join(root, 'scripts', 'install-runtime-slo-report-agent.sh'), path.join(runtimeRoot, 'scripts', 'install-runtime-slo-report-agent.sh'), ); + await fs.copyFile( + path.join(root, 'scripts', 'install-runtime-slo-soak-agent.sh'), + path.join(runtimeRoot, 'scripts', 'install-runtime-slo-soak-agent.sh'), + ); await fs.copyFile( path.join(root, 'scripts', 'install-agent-run-worker-agent.sh'), path.join(runtimeRoot, 'scripts', 'install-agent-run-worker-agent.sh'), @@ -396,6 +400,7 @@ async function writeMetadata() { ' node scripts/runtime-slo-report.mjs --write-report', ' node scripts/runtime-slo-report.mjs --write-report --prune --retention-days 30', ' bash scripts/install-runtime-slo-report-agent.sh', + ' bash scripts/install-runtime-slo-soak-agent.sh', ' node scripts/runtime-worker-drain.mjs drain goosed-3', ' node scripts/runtime-worker-drain.mjs undrain goosed-3', ' node scripts/check-tool-runtime.mjs', @@ -435,6 +440,7 @@ async function main() { await fs.chmod(path.join(runtimeRoot, 'scripts', 'install-runtime-metrics-agent.sh'), 0o755); await fs.chmod(path.join(runtimeRoot, 'scripts', 'install-runtime-heartbeat-agent.sh'), 0o755); await fs.chmod(path.join(runtimeRoot, 'scripts', 'install-runtime-slo-report-agent.sh'), 0o755); + await fs.chmod(path.join(runtimeRoot, 'scripts', 'install-runtime-slo-soak-agent.sh'), 0o755); await fs.chmod(path.join(runtimeRoot, 'scripts', 'install-agent-run-worker-agent.sh'), 0o755); await fs.chmod(path.join(runtimeRoot, 'scripts', 'agent-run-guard.mjs'), 0o755); await fs.chmod(path.join(runtimeRoot, 'scripts', 'install-agent-run-guard-agent.sh'), 0o755); diff --git a/scripts/install-runtime-slo-soak-agent.sh b/scripts/install-runtime-slo-soak-agent.sh new file mode 100755 index 0000000..93e10e1 --- /dev/null +++ b/scripts/install-runtime-slo-soak-agent.sh @@ -0,0 +1,79 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +NODE_BIN="${NODE_BIN:-/opt/homebrew/opt/node@24/bin/node}" +SCRIPT="${MEMIND_RUNTIME_SLO_SCRIPT:-$ROOT/scripts/runtime-slo-report.mjs}" +LABEL="${MEMIND_RUNTIME_SLO_SOAK_LABEL:-cn.tkmind.memind-runtime-slo-soak}" +PLIST="$HOME/Library/LaunchAgents/${LABEL}.plist" +LOG="$HOME/Library/Logs/memind-runtime-slo-soak.log" +GUI="gui/$(id -u)" +INTERVAL_SECONDS="${MEMIND_RUNTIME_SLO_SOAK_INTERVAL_SECONDS:-3600}" +REPORT_DIR="${MEMIND_RUNTIME_SLO_SOAK_REPORT_DIR:-$ROOT/reports/runtime-slo-soak}" +RETENTION_DAYS="${MEMIND_RUNTIME_SLO_SOAK_RETENTION_DAYS:-7}" + +mkdir -p "$HOME/Library/LaunchAgents" "$HOME/Library/Logs" "$REPORT_DIR" + +if [[ ! "$INTERVAL_SECONDS" =~ ^[0-9]+$ ]] || [[ "$INTERVAL_SECONDS" -lt 60 ]]; then + echo "invalid MEMIND_RUNTIME_SLO_SOAK_INTERVAL_SECONDS: $INTERVAL_SECONDS" >&2 + exit 1 +fi +if [[ ! "$RETENTION_DAYS" =~ ^[0-9]+$ ]] || [[ "$RETENTION_DAYS" -lt 1 ]]; then + echo "invalid MEMIND_RUNTIME_SLO_SOAK_RETENTION_DAYS: $RETENTION_DAYS" >&2 + exit 1 +fi +if [[ ! -x "$NODE_BIN" ]]; then + NODE_BIN="$(command -v node)" +fi +if [[ ! -f "$SCRIPT" ]]; then + echo "runtime SLO script not found: $SCRIPT" >&2 + exit 1 +fi + +cat > "$PLIST" < + + + + Label + $LABEL + ProgramArguments + + $NODE_BIN + $SCRIPT + --write-report + --prune + --retention-days + $RETENTION_DAYS + --report-dir + $REPORT_DIR + + WorkingDirectory + $ROOT + StartInterval + $INTERVAL_SECONDS + StandardOutPath + $LOG + StandardErrorPath + $LOG + EnvironmentVariables + + PATH + /opt/homebrew/bin:/opt/homebrew/opt/node@24/bin:/usr/local/bin:/usr/bin:/bin + + + +EOF + +plutil -lint "$PLIST" +launchctl bootout "$GUI/$LABEL" 2>/dev/null || true +launchctl bootstrap "$GUI" "$PLIST" +launchctl enable "$GUI/$LABEL" +launchctl kickstart -k "$GUI/$LABEL" + +echo "installed $PLIST" +echo "script: $SCRIPT" +echo "schedule: every ${INTERVAL_SECONDS}s" +echo "report_dir: $REPORT_DIR" +echo "retention_days: $RETENTION_DAYS" +echo "log: $LOG"