0dfb2e2b82
Memind CI / Test, build, and release guards (pull_request) Has been cancelled
Document the completed Docker-to-native goosed pool cutover on 103, add the production migrate script with backup/rollback paths, and extend local native pool soak/metrics helpers used as migration gates. Co-authored-by: Cursor <cursoragent@cursor.com>
366 lines
15 KiB
Markdown
366 lines
15 KiB
Markdown
# 103 生产 goosed 迁移计划:Docker/Colima → Native 多实例
|
||
|
||
> **状态:** 已执行(2026-07-30 维护窗)
|
||
> **适用范围:** 103(`58.38.22.103` / Mac Studio)生产环境
|
||
> **目标运行时:** 去掉 goosed 对 Docker/Colima 的依赖,改为 native launchd 多实例(`18006`–`18014`)
|
||
> **前提:** 允许维护窗口停机;**不要求**进行中 SSE/工具任务不断线
|
||
> **关联拓扑:** [103-runtime-topology.md](./103-runtime-topology.md)
|
||
> **发布闸门:** [production-release-guardian.md](./production-release-guardian.md)、[发包必看.md](./发包必看.md)
|
||
|
||
---
|
||
|
||
## 1. 目标与边界
|
||
|
||
### 1.1 要达成什么
|
||
|
||
| 目标 | 说明 |
|
||
|------|------|
|
||
| 去掉 goosed Docker/Colima 池 | `goosed-prod-1..9` + canary 容器退役 |
|
||
| 改为 native launchd 池 | 参考 103 已有 `tkmind_go-native` / 本机 `install-local-goosed-pool.sh` 模式,扩到 9 实例 |
|
||
| 用户数据零丢失 | RDS、PostgreSQL、MindSpace 磁盘原样保留 |
|
||
| 功能等价恢复 | 登录、聊天、续聊、页面生成、微信、Plaza/adm 等与迁移前一致 |
|
||
| 消除 bind mount inode 漂移 | Portal 发版后不再依赖 `docker compose force-recreate` remount |
|
||
|
||
### 1.2 「无缝」的定义(本计划)
|
||
|
||
| 维度 | 本计划承诺 | 不承诺 |
|
||
|------|------------|--------|
|
||
| 账号、余额、MindSpace 页面、历史落库 | 零丢失 | — |
|
||
| 系统功能 | 维护窗结束后等价恢复 | — |
|
||
| 进行中的 SSE / 工具执行 | — | 不断线;用户需刷新或重开对话 |
|
||
| 零停机 | — | 必须安排维护窗口 |
|
||
|
||
架构约束([memind-2-streaming-agent-runtime-plan.md](./architecture/memind-2-streaming-agent-runtime-plan.md)):
|
||
|
||
- 已存在 session **保持 worker affinity,不做中途迁移**
|
||
- **不做无损中途迁移**,除非 goosed 支持完整 session restore
|
||
|
||
### 1.3 不在本次范围
|
||
|
||
- 103 迁到新物理机
|
||
- 实例数从 9 缩到 2(首次迁移禁止同时缩容)
|
||
- 修改 RDS / 用户 schema
|
||
- 合并尚未走完发布闸门的 Portal 大功能(除非维护窗内单独批准发版)
|
||
|
||
---
|
||
|
||
## 2. 生产数据与依赖地图
|
||
|
||
```text
|
||
105 nginx (m.tkmind.cn)
|
||
│
|
||
▼
|
||
103 Portal :8081 cn.tkmind.memind-portal
|
||
│
|
||
├── 阿里云 RDS MySQL `goose` 用户、计费、h5_user_sessions.goosed_target、agent_run…
|
||
├── MindSpace Service :8082 /Users/john/MindSpace(不动)
|
||
├── memind_adm :8085 (不动)
|
||
├── Plaza :3001 (不动)
|
||
│
|
||
├── goosed 18006–18014 【本次替换】Docker → native
|
||
│ └── PG5432 memind_sessions(共享,不迁移数据)
|
||
│
|
||
├── PG5433 mindspace_userdata_prod
|
||
├── Redis :6379 运行时路由/指标(可清空重建)
|
||
└── 磁盘(Portal persist)
|
||
.env
|
||
/Users/john/MindSpace/data/mindspace
|
||
/Users/john/Project/Memind/users
|
||
/Users/john/Project/Memind/data
|
||
public/plaza-covers, logs, .tailscale
|
||
```
|
||
|
||
### 2.1 持久化清单(必须备份)
|
||
|
||
与 `scripts/release-portal-runtime-prod.sh` manifest 一致:
|
||
|
||
```text
|
||
.env, MindSpace, data, users, .tailscale, public/plaza-covers, logs
|
||
```
|
||
|
||
**额外必须备份(不在 persist tar 内):**
|
||
|
||
| 资产 | 备份方式 |
|
||
|------|----------|
|
||
| 阿里云 RDS `goose` | 控制台快照 + 可选逻辑导出 |
|
||
| PostgreSQL `memind_sessions` | `pg_dump` |
|
||
| PostgreSQL `mindspace_userdata_prod` | `pg_dump` |
|
||
| `goosed-prod` compose + `.env` + 镜像 tag | 目录 tar + `GOOSED_TAG` 记录 |
|
||
| `tkmind_go-native`(若已存在) | 目录拷贝 |
|
||
| Portal `.release-manifest.txt` | 记录 `release_id` / `git_head` |
|
||
| 105 nginx `m.tkmind.cn` 等配置 | 发布脚本同款备份 |
|
||
|
||
### 2.2 Session 与流量(迁移后行为不变)
|
||
|
||
- **新对话:** Portal `pickTarget()` 在 `TKMIND_API_TARGETS` 上轮询 / Redis 负载评分
|
||
- **老对话:** MySQL `h5_user_sessions.goosed_target` + Redis `session:{id}:target` 粘住原 `https://127.0.0.1:1800N`
|
||
- **goosed 会话体:** PG `memind_sessions` 多实例共享;native 直连 `127.0.0.1:5432`(替代 Docker `host.docker.internal`)
|
||
|
||
**首次迁移硬规则:native 仍监听 `18006`–`18014` 全端口**,与现网 `goosed_target` URL 兼容。
|
||
|
||
---
|
||
|
||
## 3. 迁移前准入条件(Gate)
|
||
|
||
以下 **全部满足** 才允许申请 103 维护窗:
|
||
|
||
### 3.1 本机验证(不动 103)
|
||
|
||
| # | 项 | 命令 / 标准 |
|
||
|---|-----|-------------|
|
||
| 1 | native 双实例 soak | `node scripts/soak-local-goosed-pool.mjs --minutes 30 --interval 60` 通过 |
|
||
| 2 | 池健康检查 | `node scripts/check-local-goosed-pool.mjs` |
|
||
| 3 | Portal 多 target | `.env` 中 `TKMIND_API_TARGETS=18006,18007` + 续聊 / 新建 session 冒烟 |
|
||
| 4 | native 指标 | `GOOSED_RUNTIME=native node scripts/runtime-worker-metrics.mjs status --dry-run` |
|
||
| 5 | 安装脚本 | `bash scripts/install-local-goosed-pool.sh` 可重复执行 |
|
||
| 6 | 发布脚本 native 分支 | `GOOSED_RUNTIME=native` 时 `release-portal-runtime-prod.sh` 不依赖 Docker remount(合并 main 后) |
|
||
| 7 | 回滚演练 | 本机或 103 只读:Docker compose 停/起 + `/status` 全绿 |
|
||
|
||
### 3.2 仓库与发布
|
||
|
||
| # | 项 |
|
||
|---|-----|
|
||
| 1 | 相关脚本 / 文档已合并 `main`,CI 通过 |
|
||
| 2 | `bash scripts/check-release-ready.sh` 通过 |
|
||
| 3 | 生产发布守门员 Core + Impact Gate report 已准备(与维护窗发版 commit 绑定) |
|
||
| 4 | 用户 **明确批准** 103 维护窗与迁移执行(口头「继续」不构成批准) |
|
||
|
||
### 3.3 103 只读基线(迁移前 24h 内采集)
|
||
|
||
```bash
|
||
# 只读,禁止改 103
|
||
ssh john@58.38.22.103 'cat /Users/john/Project/Memind/.release-manifest.txt'
|
||
ssh john@58.38.22.103 'grep ^TKMIND_API_TARGETS= /Users/john/Project/Memind/.env'
|
||
ssh john@58.38.22.103 'grep ^GOOSED_TAG= /Users/john/Project/goosed-prod/.env'
|
||
ssh john@58.38.22.103 'for p in $(seq 18006 18014); do curl -kfsS --connect-timeout 2 https://127.0.0.1:$p/status; echo; done'
|
||
ssh john@58.38.22.103 'curl -fsS http://127.0.0.1:8082/mindspace/v1/contract | head -c 400'
|
||
```
|
||
|
||
保存到本机 `test/_103_baselines/pre-native-migration-YYYYMMDD/`。
|
||
|
||
---
|
||
|
||
## 4. 维护窗口执行计划
|
||
|
||
**建议窗口:** ≥ 30 分钟(9 实例安装 + provider 同步 + 回归)
|
||
**建议时段:** 低峰 + 可联系 105 改 nginx 的人员在场
|
||
|
||
### Phase 0 — 备份(T0 前,可先执行)
|
||
|
||
- [ ] RDS 快照
|
||
- [ ] `pg_dump memind_sessions`、`mindspace_userdata_prod`
|
||
- [ ] 103 上执行 persist 备份(或等价 `memind-persisted-*-before.tar.gz`)
|
||
- [ ] 全目录 / compose / 105 nginx 备份
|
||
- [ ] 记录:`release_id`、`GOOSED_TAG`、九个 Docker 容器 ID、`TKMIND_API_TARGETS`
|
||
|
||
### Phase 1 — 切流量(停机开始)
|
||
|
||
- [ ] 105:维护页或 upstream 指向维护(禁止新用户进入)
|
||
- [ ] 103:可选公告 / 微信模板(「系统维护约 N 分钟」)
|
||
- [ ] **Soft drain(推荐,最多等 10 分钟):**
|
||
|
||
```bash
|
||
# 103 上,对每个 worker 设 drain,等待 active_streams 归零
|
||
# 见 memind-2-streaming-agent-runtime-plan.md P5 drain 说明
|
||
docker exec memind-runtime-redis redis-cli SET memind:runtime:worker:goosed-N:drain 1
|
||
curl -sk https://m.tkmind.cn/api/runtime/status # 观察 activeStreams
|
||
```
|
||
|
||
- [ ] 超时后 **硬停**:`cn.tkmind.memind-portal`(或仅阻止新 session,视 drain 结果)
|
||
|
||
### Phase 2 — 替换 goosed 运行时
|
||
|
||
- [ ] 停止 Docker goosed 池:
|
||
|
||
```bash
|
||
cd /Users/john/Project/goosed-prod
|
||
docker compose -f docker-compose.prod.yml down
|
||
docker rm -f goosed-prod-canary 2>/dev/null || true
|
||
```
|
||
|
||
- [ ] 部署 native 9 实例(目录模式参考 `tkmind_go-native`,端口 `18006`–`18014`):
|
||
- 同一 `GOOSE_SESSION_DB_URL` → `memind_sessions`
|
||
- `GOOSE_TLS=true`
|
||
- `GOOSED_MCP_CONTAINERIZED=0`
|
||
- `GOOSED_MCP_*` 使用宿主机路径(与 Portal `.env` 一致)
|
||
- `MINDSPACE_STORAGE_ROOT=/Users/john/MindSpace/data/mindspace`
|
||
- `PORTAL_RUNTIME_DIR=/Users/john/Project/Memind`
|
||
|
||
- [ ] 更新 Portal `.env`(若需要):
|
||
- `TKMIND_API_TARGETS=https://127.0.0.1:18006,...,https://127.0.0.1:18014`(**含 18006**)
|
||
- `GOOSED_RUNTIME=native`
|
||
- DeepSeek no-think:`127.0.0.1:18036`(去掉 `host.docker.internal` 依赖)
|
||
|
||
- [ ] **MindSpace :8082、RDS、磁盘:不移动、不删**
|
||
|
||
- [ ] Provider 同步到 **全部 9 个 target**
|
||
|
||
- [ ] 验收九个 `/status`:
|
||
|
||
```bash
|
||
for p in $(seq 18006 18014); do
|
||
echo -n "$p: "
|
||
curl -kfsS --connect-timeout 3 "https://127.0.0.1:$p/status" || echo FAIL
|
||
done
|
||
```
|
||
|
||
### Phase 3 — Portal 与附属服务
|
||
|
||
**若维护窗内无 Portal 代码变更:**
|
||
|
||
- [ ] 重启 `cn.tkmind.memind-portal`(读新 `.env`)
|
||
- [ ] **跳过** Docker remount 步骤
|
||
|
||
**若维护窗内包含 Portal runtime 发版:**
|
||
|
||
- [ ] 按 [发包必看.md](./发包必看.md) 走 canary 或晋升流程
|
||
- [ ] 发版脚本使用 `GOOSED_RUNTIME=native` 检查链
|
||
- [ ] MindSpace contract `gitSha` 与 manifest 对齐
|
||
|
||
**始终确认仍在运行:**
|
||
|
||
- [ ] `cn.tkmind.mindspace-service` :8082
|
||
- [ ] memind_adm :8085、Plaza :3001、imgproxy、SearXNG、agent-run-worker、DeepSeek no-think :18036
|
||
|
||
### Phase 4 — 功能回归(103 localhost + 105 外网)
|
||
|
||
| 优先级 | 场景 | 验证方式 |
|
||
|--------|------|----------|
|
||
| P0 | Portal `/api/status` | `curl http://127.0.0.1:8081/api/status` |
|
||
| P0 | 登录 / 鉴权 | `/auth/login`、`/auth/status` |
|
||
| P0 | 新建对话 | H5 开聊,检查 `h5_user_sessions.goosed_target` 分布 |
|
||
| P0 | **旧对话续聊** | 维护前存在的 session,迁移后 `/reply` 成功 |
|
||
| P0 | 页面列表 / 公开页 | MindSpace remote + 已有 HTML 可访问 |
|
||
| P1 | page.generate / edit_file | workspace 落盘 + `public/*.html` |
|
||
| P1 | 微信真实回调 | 非仅 health |
|
||
| P1 | MindSearch / 图片 imgproxy | 生产 URL |
|
||
| P1 | agent-run worker | 队列消费 |
|
||
| P2 | Plaza、adm 后台 | 各端口 health |
|
||
|
||
生产 Gate:维护窗发版 commit 须绑定 **Core + Impact Gate report**(见 production-release-guardian)。
|
||
|
||
### Phase 5 — 恢复流量
|
||
|
||
- [ ] 105 nginx 恢复 `58.38.22.103:8081`(或既定 upstream)
|
||
- [ ] 外网抽样:`https://m.tkmind.cn/api/status`
|
||
- [ ] 维护结束公告
|
||
|
||
### Phase 6 — 迁移后观察(24–72h)
|
||
|
||
- [ ] `GOOSED_RUNTIME=native node scripts/runtime-worker-metrics.mjs sample`
|
||
- [ ] native FD 监控(`monitor-goosed-fds.mjs`,覆盖 `18006`–`18014`)
|
||
- [ ] Redis worker 指标、SLO report 无异常尖刺
|
||
- [ ] 用户续聊 / 页面生成无集中投诉
|
||
|
||
---
|
||
|
||
## 5. 回滚计划
|
||
|
||
**触发条件:** P0 回归失败、九个 target 无法全绿、旧 session 批量无法续聊、FD 泄漏导致整机不稳定
|
||
|
||
**目标时间:** 维护窗内 < 15 分钟完成回滚
|
||
|
||
| 步骤 | 操作 |
|
||
|------|------|
|
||
| 1 | 105 维护页保持 |
|
||
| 2 | 停 native launchd 池(`18006`–`18014`) |
|
||
| 3 | `cd goosed-prod && docker compose -f docker-compose.prod.yml up -d`(**原 GOOSED_TAG**) |
|
||
| 4 | `.env` 恢复备份:`TKMIND_API_TARGETS`、Docker MCP 路径、`GOOSED_RUNTIME=docker` |
|
||
| 5 | 若 Portal 已发新版:回滚至备份 `release_id` runtime |
|
||
| 6 | 九容器 healthy + provider 同步 + `/status` 全绿 |
|
||
| 7 | 105 恢复流量 |
|
||
|
||
**回滚不需要:** 恢复 RDS / PG(迁移中若未改库)。
|
||
|
||
---
|
||
|
||
## 6. 配置对照表(Docker → Native)
|
||
|
||
| 配置项 | Docker 生产(现) | Native 生产(目标) |
|
||
|--------|-------------------|---------------------|
|
||
| goosed 进程 | `goosed-prod-1..9` 容器 | `com.tkmind.goosed-native-1800N` launchd |
|
||
| Session PG | `host.docker.internal:5432` | `127.0.0.1:5432` |
|
||
| MCP | 容器内 `/usr/local/bin/node` 等 | 宿主机 `GOOSED_MCP_*` |
|
||
| `GOOSED_MCP_CONTAINERIZED` | 隐式容器 | `0` 或不设 |
|
||
| Portal remount | 发版后 `compose force-recreate` | **不需要** |
|
||
| 监控 | `monitor-goosed-containers.mjs` | `monitor-goosed-fds.mjs`(扩展全端口) |
|
||
| 指标 | `runtime-worker-metrics` docker | `GOOSED_RUNTIME=native` |
|
||
| Canary goosed | `goosed-prod-canary:18015` | native `18015`(Phase 2 后单独迁移) |
|
||
|
||
---
|
||
|
||
## 7. 风险与缓解
|
||
|
||
| 风险 | 严重度 | 缓解 |
|
||
|------|--------|------|
|
||
| 进行中对话全部中断 | 高 | 维护公告;soft drain;恢复后提示刷新 |
|
||
| 旧 session PG restore 失败 | 高 | 迁移前用 PG 副本在 staging 验续聊 |
|
||
| provider 未覆盖全 target | 高 | 自动化检查 + 发版脚本 gate |
|
||
| MCP 路径 / sandbox 失败 | 高 | `page.generate` smoke + sandbox-fs |
|
||
| `goosed_target` 指向已下线端口 | 中 | 首次迁移保持 9 端口不变 |
|
||
| native FD 泄漏 ×9 | 中 | 72h 监控 + kickstart 阈值 |
|
||
| MindSpace / Portal 版本漂移 | 中 | contract gitSha 对账 |
|
||
| 105 切流后 CDN/缓存 | 低 | 外网真实 URL 验证 |
|
||
|
||
---
|
||
|
||
## 8. 时间线模板
|
||
|
||
| 时点 | 动作 | 环境 |
|
||
|------|------|------|
|
||
| T-14d | 本机 native 双实例 → 脚本 native 化完成 | 本机 |
|
||
| T-7d | 本机 native 9 实例 soak(若与生产同构) | 本机 |
|
||
| T-3d | 103 只读基线 + 备份演练 + Docker 回滚演练 | 103 只读/演练 |
|
||
| T-1d | 维护公告定稿;105 维护页脚本就绪 | 105 |
|
||
| T0 | Phase 0–5 执行 | 103 + 105 |
|
||
| T+24h | FD / 续聊 / 微信观察 | 103 |
|
||
| T+7d | 评审:是否退役 Colima、删 Docker 镜像、更新 topology 文档 | 文档 |
|
||
| T+30d | 可选:评估实例缩容(**单独变更**,不在本计划内) | — |
|
||
|
||
---
|
||
|
||
## 9. 迁移后文档更新
|
||
|
||
迁移成功并稳定 7 天后:
|
||
|
||
- [ ] 更新 [103-runtime-topology.md](./103-runtime-topology.md)(goosed 段改为 native 9 实例)
|
||
- [ ] 更新 [发包必看.md](./发包必看.md) §5 goosed 镜像 → native 二进制发布
|
||
- [ ] 在 [docs/branch-disposition.md](./branch-disposition.md) 登记相关功能分支(若适用)
|
||
- [ ] 归档 Docker compose 回滚包位置与 `GOOSED_TAG`
|
||
|
||
---
|
||
|
||
## 10. 本仓库工具索引
|
||
|
||
| 用途 | 路径 |
|
||
|------|------|
|
||
| 本机安装 native 池 | `bash scripts/install-local-goosed-pool.sh` |
|
||
| 池健康检查 | `node scripts/check-local-goosed-pool.mjs` |
|
||
| 长跑 soak | `node scripts/soak-local-goosed-pool.mjs` |
|
||
| native worker 指标 | `GOOSED_RUNTIME=native node scripts/runtime-worker-metrics.mjs` |
|
||
| FD 监控 | `scripts/monitor-goosed-fds.mjs` + `scripts/install-goosed-monitor.sh` |
|
||
| Portal 生产发布 | `scripts/release-portal-runtime-prod.sh` |
|
||
| 发布就绪 | `bash scripts/check-release-ready.sh` |
|
||
|
||
---
|
||
|
||
## 11. 批准记录(执行时填写)
|
||
|
||
| 字段 | 值 |
|
||
|------|-----|
|
||
| 维护窗口 | YYYY-MM-DD HH:MM – HH:MM (UTC+8) |
|
||
| 执行人 | |
|
||
| Portal `git_head` | |
|
||
| Native goosed 二进制 / release | |
|
||
| 回滚 `GOOSED_TAG` | |
|
||
| RDS 快照 ID | |
|
||
| persist_backup 路径 | |
|
||
| Gate report ID | |
|
||
| 结果 | 成功 / 回滚 |
|
||
| 备注 | |
|
||
|
||
---
|
||
|
||
**版本:** 2026-07-30 v1
|
||
**下次评审:** 本机 Phase 3(发布脚本 native 分支)合并 main 后
|