diff --git a/.gitea/workflows/ci.yml b/.gitea/workflows/ci.yml index 637b43e..a21b646 100644 --- a/.gitea/workflows/ci.yml +++ b/.gitea/workflows/ci.yml @@ -57,6 +57,9 @@ jobs: - name: Run full test suite run: npm test + - name: Verify release gate and canary routing + run: npm run test:release-gate:unit + - name: Build production frontend run: npm run build diff --git a/ENGINEERING_WORKFLOW_RULES.md b/ENGINEERING_WORKFLOW_RULES.md index d7820f7..9f541a6 100644 --- a/ENGINEERING_WORKFLOW_RULES.md +++ b/ENGINEERING_WORKFLOW_RULES.md @@ -55,7 +55,7 @@ git rebase origin/main 1. 本机不允许直接 `rsync` 到 `103` 或 `105`。 2. **禁止** SSH 登录 `105` 后直接修改业务源码(含 `scripts/wechat-mp-menu.mjs` 等);必须先本地 commit,再按发布流程上线。详见 [105 服务器变更规范](docs/105-server-operations.md)。 3. Portal 生产与测试统一走“本机构建 runtime artifact -> 打包发布”,**禁止**在 `103` 解源码包后 `npm install` / `npm run build`。 -4. Portal 唯一合法生产入口是 `bash scripts/release-portal-runtime-prod.sh`。 +4. Portal 首次生产入口是 `bash scripts/release-portal-canary-prod.sh`;它只安装并启用用户级灰度候选。`bash scripts/release-portal-runtime-prod.sh` 是整包晋升脚本,在同一候选完成灰度验收且晋升证据校验落地前继续禁止非 dry-run。 5. 发布只能从已验证的完整 `main` 打整包;分支代码必须先合并进 `main`,禁止从功能分支、单个 commit、单个修复或局部差异单包发布。 6. 发布前必须确认 CI 已通过,且没有未合并的关键变更。 7. 发布来源必须是可追溯 commit,不允许从不明工作区直接出包。 diff --git a/PRODUCTION_RELEASE_RULES.md b/PRODUCTION_RELEASE_RULES.md index fcf6678..4cb18b5 100644 --- a/PRODUCTION_RELEASE_RULES.md +++ b/PRODUCTION_RELEASE_RULES.md @@ -8,7 +8,7 @@ 2. **禁止 SSH 登录 `105` 直接修改业务代码**(含服务号菜单脚本 `scripts/wechat-mp-menu.mjs`)。105 上文件是部署产物;变更必须:本地 `test-memind` 修改 → Git commit → 正式发布 → 必要时在目标环境执行 API 同步。详见 [docs/105-server-operations.md](docs/105-server-operations.md)。 2. **Portal 生产必须是无源码 runtime 模式**:构建只发生在本机 Mac,产物是 `.runtime/portal/`;`103` 只接收 runtime artifact、继承持久目录、启动服务,**禁止**在 `103` 上 `npm install`、`npm run build` 或保留可运行源码树。 2. `MindSpace` 独立服务同样必须走单独 runtime artifact:本地 `node scripts/build-mindspace-service-runtime.mjs` -> `bash scripts/release-mindspace-service-prod.sh` -> 上传 `103` -> 备份 `/Users/john/MindSpace` 与共享 `Memind/.env` -> 原子切换到 `/Users/john/MindSpace` -> 健康检查 `127.0.0.1:8082/health` 与 `/mindspace/v1/contract`;禁止手工 SSH 改线上 `/Users/john/MindSpace` 源码。103 当前拓扑见 [docs/103-runtime-topology.md](docs/103-runtime-topology.md)。 -3. Portal 生产发布必须先构建候选 runtime 并通过用户级灰度入口;当前 `scripts/release-portal-runtime-prod.sh` 是整包替换脚本,在用户级灰度入口未上线、未验证前禁止调用。灰度只能命中明确的不可变用户身份,未命中请求必须继续走稳定版本,禁止直接全量切换。 +3. Portal 生产发布必须先构建候选 runtime,并只通过 `scripts/release-portal-canary-prod.sh` 启用用户级灰度。灰度只能命中明确的不可变用户身份,未命中、身份解析失败或候选不健康必须继续走稳定版本。`scripts/release-portal-runtime-prod.sh` 仍是整包替换脚本,在同一候选的灰度验收和晋升证据校验完成前禁止非 dry-run。 4. `scripts/release-prod.sh`(源码包发布)已停用,不得再用于 Portal;`rsync_to_server.sh` 与任何面向 `105` 的直接同步脚本也只保留为禁用提示。 5. Portal 发布包不得携带运行态资产;`.env`、`data/`、`users/`、`.tailscale/`、`public/plaza-covers/`、`logs/` 只能从线上现有 live 目录继承。`/Users/john/MindSpace` 是独立 MindSpace Service 的生产根目录,不属于 Portal runtime 包;`/Users/john/Project/Memind/MindSpace` 只允许作为旧链路兼容/存量目录处理,不得再被写成 MindSpace Service 的当前根目录。 6. runtime artifact 必须包含 `server.mjs` 与 `mindspace-sandbox-mcp.mjs`(`sandbox-fs` 扩展依赖的独立子进程入口)。 diff --git a/docs/103-runtime-topology.md b/docs/103-runtime-topology.md index f66b1f5..0d2ad56 100644 --- a/docs/103-runtime-topology.md +++ b/docs/103-runtime-topology.md @@ -35,6 +35,19 @@ Local Colima may exist for other workloads, but local Memind/goosed is not suppo Portal must be released as a runtime artifact. Do not edit source directly on 103. +### User-level canary (when enabled) + +| Item | Value | +|------|-------| +| Identity router | `127.0.0.1:18080` | +| Candidate Portal | `127.0.0.1:18081` | +| Candidate goosed | `127.0.0.1:18015` | +| Release entry | `scripts/release-portal-canary-prod.sh` | +| Rollback entry | `scripts/rollback-portal-canary-prod.sh` | + +The stable Portal stays on 8081. The 105 reverse tunnel points to 18080 only after the +candidate, router, backups, immutable identity checks, and fallback checks pass. + ## MindSpace Service | Item | Value | diff --git a/docs/production-release-guardian.md b/docs/production-release-guardian.md index 9637812..22c5bb7 100644 --- a/docs/production-release-guardian.md +++ b/docs/production-release-guardian.md @@ -544,9 +544,18 @@ failed = skipped = blocked = unknown = cleanup_failed = 0 ### 10.0 用户级灰度前置条件 -当前 Portal 发布脚本会替换 8081 live 目录,不具备版本级用户路由能力,因此在用户级灰度代理和候选 runtime 并行运行方案完成前,禁止调用该脚本执行生产发布。灰度身份必须使用不可变用户 ID/用户名/微信绑定 ID;禁止按显示昵称(包括“唐”)匹配。未命中或身份解析失败必须回落稳定版本,候选不可用必须自动回退。 +用户级灰度入口由 `scripts/release-portal-canary-prod.sh` 安装:稳定 Portal 保持 8081, +候选 Portal 使用 18081,候选 goosed 使用 18015,身份路由器使用 18080。灰度身份必须 +使用不可变用户 ID、唯一用户名或微信绑定账号标识;禁止按显示昵称(包括“唐”)匹配。未命中、 +身份解析失败或候选不可用必须自动回落稳定版本。 -本地路由契约见 [docs/release-canary-103.md](release-canary-103.md),对应测试为 `release-gate/canary-routing.test.mjs`。该测试通过只代表身份匹配契约通过,不代表 103 已具备灰度入口。 +完整契约和回滚入口见 [103 用户级灰度发布](release-canary-103.md),对应测试至少包括 +`release-gate/canary-routing.test.mjs`、`memind-canary-proxy.test.mjs` 和 +`release-gate/release-script.test.mjs`。测试通过不代表已经部署;103 灰度仍须绑定同一 +commit、artifact 和 Gate report,并完成备份、只读预检及人工批准。 + +`scripts/release-portal-runtime-prod.sh` 会替换 8081 live 目录。它只用于同一候选完成灰度 +验收后的全量晋升;在晋升证据校验完成前继续硬阻断非 dry-run,不能拿灰度入口替代晋升批准。 `scripts/release-portal-runtime-prod.sh` 必须按以下顺序执行: diff --git a/docs/release-canary-103.md b/docs/release-canary-103.md index 5632ddd..f4d9e7f 100644 --- a/docs/release-canary-103.md +++ b/docs/release-canary-103.md @@ -1,13 +1,71 @@ -# 103 用户级灰度发布设计 +# 103 用户级灰度发布 -当前 `scripts/release-portal-runtime-prod.sh` 是整包替换脚本,不能用于“只让 john 和精确微信用户唐”灰度。因此在闸门通过前禁止调用它进行生产发布。 +Portal 的首个生产动作必须是用户级灰度,不能直接整包替换稳定版本。 -候选版本必须与稳定版本并行运行,由入口代理依据登录后的不可变身份选择版本: +## 运行拓扑 -- `john`:按唯一用户名或数据库用户 ID 匹配。 -- 微信用户“唐”:按绑定的不可变用户 ID `wx_ul610et8` 匹配;禁止按昵称匹配,避免与其他同名用户串流。 -- 未命中、身份缺失、解析异常:全部回落到稳定版本。 +| 组件 | 地址 | 说明 | +|---|---|---| +| 稳定 Portal | `127.0.0.1:8081` | 保持当前 live 目录和进程,不在灰度发布中替换 | +| 灰度路由器 | `127.0.0.1:18080` | 105 的 `19081` 反向隧道在全部检查通过后切到这里 | +| 候选 Portal | `127.0.0.1:18081` | 独立、无源码 runtime;使用被动运行时模式 | +| 候选 goosed | `127.0.0.1:18015` | 独立容器并挂载候选 runtime,不重建稳定 9 个容器 | -候选版本必须使用独立监听端口和独立 runtime 目录,但共享经审计的持久数据与数据库连接;灰度入口必须支持 SSE/附件/公开页请求的完整转发,并保留稳定版本回退。 +候选 Portal 通过 `MEMIND_PORTAL_H5_ROOT` 共享经审计的持久数据和数据库,但 +`MEMIND_RUNTIME_ROLE=candidate` 会禁止重复启动 MindSpace、Plaza、提醒、订阅续期和 +Agent 恢复扫描等单例后台任务。 -上线前必须在本地证明:命中用户只到候选版本、其他用户只到稳定版本、同名非目标用户不命中、身份解析失败回落稳定、候选不可用自动回退。生产发布仍需在 `main` 干净提交、187 场景全通过并获得单独的生产灰度确认后执行;本设计不会触发生产动作。 +## 身份和回落规则 + +- `john`:按数据库中的唯一用户名或用户 ID 匹配。 +- 微信用户“唐”:只按绑定微信身份的唯一账号标识 `wx_ul610et8` 匹配;该值在生产库中是唯一 `username/slug`,不是显示昵称。 +- 显示昵称永远不参与路由。 +- Cookie 会话通过 `h5_login_sessions` 的 token hash 只读解析。 +- 微信回调通过 `FromUserName` 和 `h5_user_wechat_identities` 只读解析到绑定账号,再用唯一账号标识匹配。 +- 未命中、无身份、过期/撤销会话、数据库异常:走稳定版本。 +- 候选 `/api/status` 不健康:自动走稳定版本。 +- SSE、附件、公开页和普通 API 均由同一反向代理转发;微信 XML body 必须原样保留。 + +## 发布入口 + +候选必须来自干净且等于 `origin/main` 的完整 `main`,CI 成功,并具有与 +`.runtime/portal` 完全一致、未过期的 187 项 Gate report。 + +```bash +bash scripts/release-portal-canary-prod.sh --yes +``` + +该脚本按顺序执行: + +1. 验证源码、分支、远端主线、Gate report 和 artifact hash。 +2. 执行 103 只读预检。 +3. 上传经过验证的候选包。 +4. 创建稳定 live 全量备份和持久目录备份,并校验 SHA256、gzip 和关键目录。 +5. 启动独立候选 goosed、候选 Portal 和灰度路由器。 +6. 验证不可变身份配置、稳定/候选健康和路由器诊断。 +7. 短暂阻止新 Agent Run,排空活动任务。 +8. 只把反向隧道从 8081 切到 18080。 +9. 主动停止候选,证明目标身份会自动回落稳定版,再恢复候选。 + +任一步失败都会把隧道恢复到稳定 8081,停止候选服务并保留候选目录和备份证据。 + +## 回滚 + +```bash +bash scripts/rollback-portal-canary-prod.sh --yes +``` + +回滚只执行以下动作: + +- 把 105 反向隧道恢复到稳定 Portal 8081。 +- 停止灰度路由器、候选 Portal 和候选 goosed。 +- 清除发布排水标记。 +- 验证稳定 Portal 和 105 隧道健康。 + +不会删除候选目录、备份或稳定运行目录。 + +## 全量晋升 + +`scripts/release-portal-runtime-prod.sh` 仍是整包替换脚本,不能把“灰度路由器已实现” +解释为自动获得全量切换权限。全量晋升必须使用同一 commit 和 artifact,保留灰度验收 +证据,并再次取得明确批准;在晋升入口完成对应证据校验前,该脚本继续硬阻断非 dry-run。 diff --git a/docs/发包必看.md b/docs/发包必看.md index 18550b2..6fc68f9 100644 --- a/docs/发包必看.md +++ b/docs/发包必看.md @@ -45,13 +45,15 @@ npm run check:mindspace-public-links ## 2. Portal runtime 发布唯一流程 -生产发布守门员落地、完整 Gate report 通过且获得本次明确人工批准后,只使用: +生产发布守门员、完整 Gate report 和本次明确人工批准全部满足后,首次生产动作只使用: ```bash -bash scripts/release-portal-runtime-prod.sh --yes +bash scripts/release-portal-canary-prod.sh --yes ``` -生产 `103` 禁止使用 `--skip-tests` 或任何范围绕过变量。本文历史记录中的旧绕过命令只用于事故复盘,不构成当前发布授权。 +生产 `103` 禁止使用 `--skip-tests` 或任何范围绕过变量。整包替换脚本 +`release-portal-runtime-prod.sh` 只用于同一候选完成灰度验收后的全量晋升,在晋升证据 +校验完成前仍会硬阻断。本文历史记录中的旧绕过命令只用于事故复盘,不构成当前发布授权。 这个脚本必须完成这些动作: diff --git a/memind-canary-proxy.mjs b/memind-canary-proxy.mjs new file mode 100644 index 0000000..e7f8754 --- /dev/null +++ b/memind-canary-proxy.mjs @@ -0,0 +1,569 @@ +#!/usr/bin/env node +import crypto from 'node:crypto'; +import http from 'node:http'; +import { pathToFileURL } from 'node:url'; + +import mysql from 'mysql2/promise'; + +import { + canaryPolicyFromEnv, + resolveCanaryTarget, +} from './release-gate/canary-routing.mjs'; +import { loadMemindEnvFiles } from './scripts/memind-runtime-profile.mjs'; + +const SESSION_COOKIE = 'tkmind_user_session'; +const DEFAULT_BODY_LIMIT = 2 * 1024 * 1024; +const RETRYABLE_CONNECT_ERRORS = new Set([ + 'ECONNREFUSED', + 'EHOSTUNREACH', + 'ENETUNREACH', +]); + +function csvValues(value) { + return String(value ?? '') + .split(',') + .map((item) => item.trim()) + .filter(Boolean); +} + +export function parseCookieHeader(header) { + const cookies = new Map(); + for (const item of String(header ?? '').split(';')) { + const separator = item.indexOf('='); + if (separator < 1) continue; + const name = item.slice(0, separator).trim(); + const rawValue = item.slice(separator + 1).trim(); + if (!name) continue; + try { + cookies.set(name, decodeURIComponent(rawValue)); + } catch { + cookies.set(name, rawValue); + } + } + return cookies; +} + +export function parseWechatOpenid(xml) { + const match = String(xml ?? '').match( + /<\/FromUserName>/i, + ); + return String(match?.[1] ?? '').trim(); +} + +function sessionTokenHash(token) { + return crypto.createHash('sha256').update(token).digest('hex'); +} + +function normalizeLoopbackTarget(value, label) { + const target = new URL(String(value ?? '')); + if ( + target.protocol !== 'http:' + || !['127.0.0.1', 'localhost', '::1'].includes(target.hostname) + ) { + throw new Error(`${label} must be a loopback http URL`); + } + target.pathname = target.pathname.replace(/\/+$/, ''); + target.search = ''; + target.hash = ''; + return target; +} + +function createIdentityPool(env = process.env) { + const poolOptions = { + waitForConnections: true, + connectionLimit: Math.max( + 1, + Math.min(8, Number(env.MEMIND_CANARY_DB_POOL_SIZE ?? 4)), + ), + queueLimit: 32, + }; + if (env.DATABASE_URL) { + return mysql.createPool({ uri: env.DATABASE_URL, ...poolOptions }); + } + if (!env.MYSQL_HOST || !env.MYSQL_DATABASE) { + throw new Error('Canary identity routing requires DATABASE_URL or MYSQL_*'); + } + return mysql.createPool({ + host: env.MYSQL_HOST, + port: Number(env.MYSQL_PORT ?? 3306), + user: env.MYSQL_USER ?? 'boot', + password: env.MYSQL_PASSWORD ?? '', + database: env.MYSQL_DATABASE, + ...poolOptions, + }); +} + +export function createCanaryIdentityResolver({ + pool, + appId = '', + now = () => Date.now(), +} = {}) { + if (!pool?.query) throw new Error('identity resolver requires a database pool'); + + return { + async fromSessionToken(token) { + if (!token) return null; + const [rows] = await pool.query( + `SELECT s.user_id, u.username + FROM h5_login_sessions s + JOIN h5_users u ON u.id = s.user_id + WHERE s.token_hash = ? + AND s.revoked_at IS NULL + AND s.expires_at > ? + AND u.status <> 'disabled' + LIMIT 1`, + [sessionTokenHash(token), now()], + ); + const row = rows[0]; + return row + ? { + id: String(row.user_id ?? ''), + username: String(row.username ?? ''), + } + : null; + }, + + async fromWechatOpenid(openid) { + if (!openid || !appId) return null; + const [rows] = await pool.query( + `SELECT wi.user_id, u.username + FROM h5_user_wechat_identities wi + JOIN h5_users u ON u.id = wi.user_id + WHERE wi.app_id = ? + AND wi.openid = ? + AND u.status <> 'disabled' + LIMIT 1`, + [appId, openid], + ); + const row = rows[0]; + return row + ? { + id: String(row.user_id ?? ''), + username: String(row.username ?? ''), + wechatUserId: String(row.username ?? ''), + } + : null; + }, + }; +} + +export async function verifyCanaryPolicySelectors(pool, policy) { + const selectors = [ + ...[...(policy.userIds ?? [])].map((value) => ['id', value]), + ...[...(policy.usernames ?? [])].map((value) => ['username', value]), + ...[...(policy.wechatUserIds ?? [])].map((value) => ['wechat', value]), + ]; + for (const [kind, value] of selectors) { + if (kind === 'username') { + const [rows] = await pool.query( + `SELECT id FROM h5_users + WHERE LOWER(username) = LOWER(?) AND status <> 'disabled' + LIMIT 2`, + [value], + ); + if (rows.length !== 1) { + throw new Error('configured canary username does not resolve uniquely'); + } + continue; + } + const lookupColumn = kind === 'wechat' ? 'u.username' : 'u.id'; + const [rows] = await pool.query( + `SELECT u.id, + EXISTS( + SELECT 1 FROM h5_user_wechat_identities wi WHERE wi.user_id = u.id + ) AS has_wechat + FROM h5_users u + WHERE ${lookupColumn} = ? AND u.status <> 'disabled' + LIMIT 2`, + [value], + ); + if (rows.length !== 1) { + throw new Error('configured canary immutable user selector does not resolve uniquely'); + } + if (kind === 'wechat' && Number(rows[0].has_wechat ?? 0) !== 1) { + throw new Error('configured WeChat canary user has no bound immutable identity'); + } + } + return { + userIds: policy.userIds?.size ?? 0, + usernames: policy.usernames?.size ?? 0, + wechatUserIds: policy.wechatUserIds?.size ?? 0, + }; +} + +function requestBody(req, limit = DEFAULT_BODY_LIMIT) { + return new Promise((resolve, reject) => { + const chunks = []; + let bytes = 0; + req.on('data', (chunk) => { + bytes += chunk.length; + if (bytes > limit) { + reject(Object.assign(new Error('request body too large'), { statusCode: 413 })); + req.destroy(); + return; + } + chunks.push(chunk); + }); + req.on('end', () => resolve(Buffer.concat(chunks))); + req.on('error', reject); + }); +} + +function healthRequest(target, timeoutMs, expectedRole = '') { + return new Promise((resolve) => { + const request = http.request( + { + hostname: target.hostname, + port: target.port, + method: 'GET', + path: `${target.pathname}/api/status`.replace(/\/{2,}/g, '/'), + headers: { connection: 'close' }, + }, + (response) => { + response.resume(); + response.once('end', () => { + const runtimeRole = String( + response.headers['x-memind-runtime-role'] ?? '', + ).trim().toLowerCase(); + resolve( + response.statusCode === 200 + && (!expectedRole || runtimeRole === expectedRole), + ); + }); + }, + ); + request.setTimeout(timeoutMs, () => request.destroy()); + request.once('error', () => resolve(false)); + request.end(); + }); +} + +function appendForwardedFor(current, remoteAddress) { + const values = [String(current ?? '').trim(), String(remoteAddress ?? '').trim()] + .filter(Boolean); + return values.join(', '); +} + +export function createMemindCanaryProxy({ + stableTarget, + candidateTarget, + policy, + identityResolver, + diagnosticSecret, + candidateHealthTtlMs = 1_000, + candidateHealthTimeoutMs = 750, + logger = console, +} = {}) { + const stable = normalizeLoopbackTarget(stableTarget, 'stable target'); + const candidate = normalizeLoopbackTarget(candidateTarget, 'candidate target'); + if (!policy) throw new Error('canary proxy requires a routing policy'); + if (!identityResolver) throw new Error('canary proxy requires an identity resolver'); + if (!diagnosticSecret) throw new Error('canary proxy requires a diagnostic secret'); + + let healthCache = { checkedAt: 0, healthy: false }; + let healthPromise = null; + + const candidateHealthy = async ({ force = false } = {}) => { + const currentTime = Date.now(); + if (!force && currentTime - healthCache.checkedAt < candidateHealthTtlMs) { + return healthCache.healthy; + } + if (!healthPromise) { + healthPromise = healthRequest(candidate, candidateHealthTimeoutMs, 'candidate') + .then((healthy) => { + healthCache = { checkedAt: Date.now(), healthy }; + return healthy; + }) + .finally(() => { + healthPromise = null; + }); + } + return healthPromise; + }; + + const stableHealthy = () => healthRequest(stable, candidateHealthTimeoutMs); + + const resolveIdentity = async (req, bufferedBody) => { + if ( + req.method === 'POST' + && String(req.url ?? '').split('?')[0] === '/webhooks/wechat-mp/messages' + ) { + return identityResolver.fromWechatOpenid( + parseWechatOpenid(bufferedBody?.toString('utf8')), + ); + } + const token = parseCookieHeader(req.headers.cookie).get(SESSION_COOKIE); + return identityResolver.fromSessionToken(token); + }; + + const selectTarget = async (req, bufferedBody) => { + let identity = null; + try { + identity = await resolveIdentity(req, bufferedBody); + } catch (error) { + logger.warn?.( + 'Canary identity resolution failed closed to stable:', + error instanceof Error ? error.message : String(error), + ); + return { route: 'stable', target: stable }; + } + if (resolveCanaryTarget(identity, policy) !== 'candidate') { + return { route: 'stable', target: stable }; + } + if (!(await candidateHealthy())) { + return { route: 'stable', target: stable }; + } + return { route: 'candidate', target: candidate }; + }; + + const proxyRequest = (req, res, selection, bufferedBody, allowFallback = true) => { + const headers = { + ...req.headers, + 'x-forwarded-for': appendForwardedFor( + req.headers['x-forwarded-for'], + req.socket.remoteAddress, + ), + 'x-memind-canary-route': selection.route, + }; + delete headers['x-memind-runtime-route']; + if (bufferedBody) { + headers['content-length'] = String(bufferedBody.length); + delete headers['transfer-encoding']; + } + const hasDeclaredBody = + Boolean(req.headers['content-length']) + || Boolean(req.headers['transfer-encoding']); + const safeToReplay = + Boolean(bufferedBody) + || ['GET', 'HEAD', 'OPTIONS'].includes(String(req.method ?? '').toUpperCase()) + || !hasDeclaredBody; + + let connected = false; + const upstream = http.request( + { + hostname: selection.target.hostname, + port: selection.target.port, + method: req.method, + path: `${selection.target.pathname}${req.url ?? '/'}`.replace(/\/{2,}/g, '/'), + headers, + }, + (upstreamResponse) => { + const responseHeaders = { + ...upstreamResponse.headers, + 'x-memind-runtime-route': selection.route, + }; + res.writeHead(upstreamResponse.statusCode ?? 502, responseHeaders); + upstreamResponse.pipe(res); + }, + ); + upstream.on('socket', (socket) => { + socket.once('connect', () => { + connected = true; + }); + }); + upstream.once('error', (error) => { + if ( + allowFallback + && selection.route === 'candidate' + && safeToReplay + && !connected + && RETRYABLE_CONNECT_ERRORS.has(error?.code) + && !res.headersSent + ) { + healthCache = { checkedAt: Date.now(), healthy: false }; + proxyRequest( + req, + res, + { route: 'stable', target: stable }, + bufferedBody, + false, + ); + return; + } + if (!res.headersSent) { + res.writeHead(502, { 'content-type': 'application/json; charset=utf-8' }); + } + res.end(JSON.stringify({ ok: false, message: 'canary upstream unavailable' })); + }); + if (bufferedBody) { + upstream.end(bufferedBody); + } else if (!hasDeclaredBody) { + upstream.end(); + } else { + req.pipe(upstream); + } + }; + + const server = http.createServer(async (req, res) => { + try { + const requestUrl = new URL(req.url ?? '/', 'http://127.0.0.1'); + const pathname = requestUrl.pathname; + if (pathname === '/__memind_canary/health') { + if (req.headers['x-memind-canary-secret'] !== diagnosticSecret) { + res.writeHead(404).end(); + return; + } + const [stableOk, candidateOk] = await Promise.all([ + stableHealthy(), + candidateHealthy({ force: true }), + ]); + res.writeHead(stableOk ? 200 : 503, { + 'content-type': 'application/json; charset=utf-8', + 'cache-control': 'no-store', + }); + res.end(JSON.stringify({ + ok: stableOk, + stable: stableOk, + candidate: candidateOk, + policy: { + userIds: policy.userIds?.size ?? 0, + usernames: policy.usernames?.size ?? 0, + wechatUserIds: policy.wechatUserIds?.size ?? 0, + }, + })); + return; + } + if (pathname === '/__memind_canary/route-probe') { + if (req.headers['x-memind-canary-secret'] !== diagnosticSecret) { + res.writeHead(404).end(); + return; + } + const identity = { + id: requestUrl.searchParams.get('id') ?? '', + username: requestUrl.searchParams.get('username') ?? '', + wechatUserId: requestUrl.searchParams.get('wechat_user_id') ?? '', + }; + const configuredTarget = resolveCanaryTarget(identity, policy); + const candidateOk = + configuredTarget === 'candidate' ? await candidateHealthy({ force: true }) : false; + const route = + configuredTarget === 'candidate' && candidateOk ? 'candidate' : 'stable'; + res.writeHead(200, { + 'content-type': 'application/json; charset=utf-8', + 'cache-control': 'no-store', + }); + res.end(JSON.stringify({ + route, + configuredTarget, + candidate: candidateOk, + })); + return; + } + + const needsBody = + req.method === 'POST' + && pathname === '/webhooks/wechat-mp/messages'; + const bufferedBody = needsBody ? await requestBody(req) : null; + const selection = await selectTarget(req, bufferedBody); + proxyRequest(req, res, selection, bufferedBody); + } catch (error) { + const statusCode = Number(error?.statusCode ?? 500); + if (!res.headersSent) { + res.writeHead(statusCode, { 'content-type': 'application/json; charset=utf-8' }); + } + res.end(JSON.stringify({ ok: false, message: 'canary routing failed closed' })); + } + }); + + server.on('upgrade', async (req, socket, head) => { + let selection; + try { + selection = await selectTarget(req, null); + } catch { + selection = { route: 'stable', target: stable }; + } + const headers = { + ...req.headers, + connection: 'Upgrade', + upgrade: req.headers.upgrade ?? 'websocket', + 'x-memind-canary-route': selection.route, + }; + const upstream = http.request({ + hostname: selection.target.hostname, + port: selection.target.port, + method: req.method, + path: req.url, + headers, + }); + upstream.on('upgrade', (response, upstreamSocket, upstreamHead) => { + const statusLine = + `HTTP/1.1 ${response.statusCode ?? 101} ${response.statusMessage ?? 'Switching Protocols'}\r\n`; + const responseHeaders = Object.entries(response.headers) + .flatMap(([name, value]) => { + const values = Array.isArray(value) ? value : [value]; + return values.filter((item) => item != null).map((item) => `${name}: ${item}\r\n`); + }) + .join(''); + socket.write(`${statusLine}${responseHeaders}\r\n`); + if (upstreamHead.length) socket.write(upstreamHead); + if (head.length) upstreamSocket.write(head); + upstreamSocket.pipe(socket); + socket.pipe(upstreamSocket); + }); + upstream.once('error', () => socket.destroy()); + upstream.end(); + }); + + return { + server, + candidateHealthy, + stableHealthy, + }; +} + +async function main() { + const persistentRoot = + process.env.MEMIND_PORTAL_H5_ROOT + || process.env.MEMIND_CANARY_STABLE_ROOT + || process.cwd(); + loadMemindEnvFiles(persistentRoot); + const policy = canaryPolicyFromEnv(process.env); + const pool = createIdentityPool(process.env); + const selectorSummary = await verifyCanaryPolicySelectors(pool, policy); + if (process.argv.includes('--check')) { + console.log(JSON.stringify({ ok: true, policy: selectorSummary })); + await pool.end(); + return; + } + + const identityResolver = createCanaryIdentityResolver({ + pool, + appId: + process.env.H5_WECHAT_MP_APP_ID + || process.env.H5_WECHAT_APP_ID + || '', + }); + const listenHost = process.env.MEMIND_CANARY_PROXY_HOST || '127.0.0.1'; + const listenPort = Number(process.env.MEMIND_CANARY_PROXY_PORT ?? 18080); + const proxy = createMemindCanaryProxy({ + stableTarget: process.env.MEMIND_CANARY_STABLE_URL || 'http://127.0.0.1:8081', + candidateTarget: + process.env.MEMIND_CANARY_CANDIDATE_URL || 'http://127.0.0.1:18081', + policy, + identityResolver, + diagnosticSecret: process.env.MEMIND_CANARY_DIAGNOSTIC_SECRET, + }); + + const shutdown = async () => { + proxy.server.close(); + await pool.end().catch(() => {}); + }; + process.once('SIGTERM', shutdown); + process.once('SIGINT', shutdown); + proxy.server.listen(listenPort, listenHost, () => { + console.log( + `Memind canary proxy listening on http://${listenHost}:${listenPort} ` + + `(stable=8081 candidate=18081 selectors=${JSON.stringify(selectorSummary)})`, + ); + }); +} + +if ( + process.argv[1] + && import.meta.url === pathToFileURL(process.argv[1]).href +) { + main().catch((error) => { + console.error(error instanceof Error ? error.message : String(error)); + process.exit(1); + }); +} diff --git a/memind-canary-proxy.test.mjs b/memind-canary-proxy.test.mjs new file mode 100644 index 0000000..60af70d --- /dev/null +++ b/memind-canary-proxy.test.mjs @@ -0,0 +1,397 @@ +import assert from 'node:assert/strict'; +import http from 'node:http'; +import test from 'node:test'; + +import { createCanaryPolicy } from './release-gate/canary-routing.mjs'; +import { + createCanaryIdentityResolver, + createMemindCanaryProxy, + parseCookieHeader, + parseWechatOpenid, + verifyCanaryPolicySelectors, +} from './memind-canary-proxy.mjs'; + +function listen(server) { + return new Promise((resolve, reject) => { + server.once('error', reject); + server.listen(0, '127.0.0.1', () => { + const address = server.address(); + resolve(`http://127.0.0.1:${address.port}`); + }); + }); +} + +function close(server) { + return new Promise((resolve) => server.close(resolve)); +} + +function request(baseUrl, { + method = 'GET', + path = '/', + headers = {}, + body = '', +} = {}) { + const target = new URL(path, baseUrl); + return new Promise((resolve, reject) => { + const req = http.request( + { + hostname: target.hostname, + port: target.port, + path: `${target.pathname}${target.search}`, + method, + headers: { + ...headers, + ...(body ? { 'content-length': Buffer.byteLength(body) } : {}), + }, + }, + (res) => { + const chunks = []; + res.on('data', (chunk) => chunks.push(chunk)); + res.on('end', () => resolve({ + status: res.statusCode, + headers: res.headers, + body: Buffer.concat(chunks).toString('utf8'), + })); + }, + ); + req.once('error', reject); + req.end(body); + }); +} + +function upstream(name) { + return http.createServer((req, res) => { + if (req.url === '/api/status') { + res.writeHead(200, { 'x-memind-runtime-role': name }); + res.end('ok'); + return; + } + const chunks = []; + req.on('data', (chunk) => chunks.push(chunk)); + req.on('end', () => { + res.writeHead(200, { 'content-type': 'text/plain' }); + res.end(`${name}:${Buffer.concat(chunks).toString('utf8')}`); + }); + }); +} + +function candidateThatStopsAfterHealth() { + const server = http.createServer((req, res) => { + if (req.url !== '/api/status') { + res.writeHead(200); + res.end('candidate'); + return; + } + server.close(); + res.writeHead(200, { 'x-memind-runtime-role': 'candidate' }); + res.end('ok'); + }); + return server; +} + +test('cookie and WeChat parsers preserve immutable routing inputs', () => { + const cookies = parseCookieHeader('a=1; tkmind_user_session=abc%2D123'); + assert.equal(cookies.get('tkmind_user_session'), 'abc-123'); + assert.equal( + parseWechatOpenid( + '', + ), + 'openid-target', + ); + assert.equal(parseWechatOpenid(''), ''); +}); + +test('identity resolver hashes sessions and maps WeChat openid to bound user id', async () => { + const calls = []; + const resolver = createCanaryIdentityResolver({ + pool: { + async query(sql, params) { + calls.push({ sql, params }); + if (sql.includes('h5_login_sessions')) { + return [[{ user_id: 'user-john', username: 'john' }]]; + } + return [[{ user_id: 'wx_ul610et8', username: 'wechat_user' }]]; + }, + }, + appId: 'app-1', + now: () => 123, + }); + + assert.deepEqual(await resolver.fromSessionToken('token-1'), { + id: 'user-john', + username: 'john', + }); + assert.match(calls[0].params[0], /^[a-f0-9]{64}$/); + assert.equal(calls[0].params[1], 123); + + assert.deepEqual(await resolver.fromWechatOpenid('openid-1'), { + id: 'wx_ul610et8', + username: 'wechat_user', + wechatUserId: 'wechat_user', + }); + assert.deepEqual(calls[1].params, ['app-1', 'openid-1']); +}); + +test('WeChat canary selector validates the bound unique username, not a production UUID', async () => { + const calls = []; + const summary = await verifyCanaryPolicySelectors( + { + async query(sql, params) { + calls.push({ sql, params }); + if (sql.includes('LOWER(username)')) return [[{ id: 'john-uuid' }]]; + return [[{ id: 'wechat-uuid', has_wechat: 1 }]]; + }, + }, + createCanaryPolicy({ + usernames: ['john'], + wechatUserIds: ['wx_ul610et8'], + }), + ); + + assert.deepEqual(summary, { userIds: 0, usernames: 1, wechatUserIds: 1 }); + assert.match(calls[1].sql, /WHERE u\.username = \?/); + assert.deepEqual(calls[1].params, ['wx_ul610et8']); +}); + +test('proxy routes only immutable canary identities and preserves WeChat bodies', async (t) => { + const stableServer = upstream('stable'); + const candidateServer = upstream('candidate'); + const stableUrl = await listen(stableServer); + const candidateUrl = await listen(candidateServer); + const policy = createCanaryPolicy({ + usernames: ['john'], + wechatUserIds: ['wx_ul610et8'], + }); + const identityResolver = { + async fromSessionToken(token) { + if (token === 'john-token') return { id: 'user-john', username: 'john' }; + if (token === 'same-name-token') { + return { id: 'other-user', username: 'other', displayName: '唐' }; + } + return null; + }, + async fromWechatOpenid(openid) { + if (openid === 'openid-target') { + return { + id: 'wechat-user-uuid', + username: 'wx_ul610et8', + wechatUserId: 'wx_ul610et8', + }; + } + return null; + }, + }; + const proxy = createMemindCanaryProxy({ + stableTarget: stableUrl, + candidateTarget: candidateUrl, + policy, + identityResolver, + diagnosticSecret: 'diagnostic-secret', + candidateHealthTtlMs: 0, + }); + const proxyUrl = await listen(proxy.server); + t.after(async () => { + await close(proxy.server); + await close(stableServer); + if (candidateServer.listening) await close(candidateServer); + }); + + const john = await request(proxyUrl, { + headers: { cookie: 'tkmind_user_session=john-token' }, + }); + assert.equal(john.body, 'candidate:'); + assert.equal(john.headers['x-memind-runtime-route'], 'candidate'); + + const missingIdentity = await request(proxyUrl); + assert.equal(missingIdentity.body, 'stable:'); + assert.equal(missingIdentity.headers['x-memind-runtime-route'], 'stable'); + + const sameNickname = await request(proxyUrl, { + headers: { cookie: 'tkmind_user_session=same-name-token' }, + }); + assert.equal(sameNickname.body, 'stable:'); + + const xml = + '' + + ''; + const wechat = await request(proxyUrl, { + method: 'POST', + path: '/webhooks/wechat-mp/messages', + headers: { 'content-type': 'text/xml' }, + body: xml, + }); + assert.equal(wechat.body, `candidate:${xml}`); + assert.equal(wechat.headers['x-memind-runtime-route'], 'candidate'); + + const health = await request(proxyUrl, { + path: '/__memind_canary/health', + headers: { 'x-memind-canary-secret': 'diagnostic-secret' }, + }); + assert.equal(health.status, 200); + assert.deepEqual(JSON.parse(health.body), { + ok: true, + stable: true, + candidate: true, + policy: { userIds: 0, usernames: 1, wechatUserIds: 1 }, + }); + + await close(candidateServer); + const routeProbe = await request(proxyUrl, { + path: '/__memind_canary/route-probe?username=john', + headers: { 'x-memind-canary-secret': 'diagnostic-secret' }, + }); + assert.deepEqual(JSON.parse(routeProbe.body), { + route: 'stable', + configuredTarget: 'candidate', + candidate: false, + }); + const fallback = await request(proxyUrl, { + headers: { cookie: 'tkmind_user_session=john-token' }, + }); + assert.equal(fallback.body, 'stable:'); + assert.equal(fallback.headers['x-memind-runtime-route'], 'stable'); +}); + +test('identity lookup errors fail closed to stable', async (t) => { + const stableServer = upstream('stable'); + const candidateServer = upstream('candidate'); + const stableUrl = await listen(stableServer); + const candidateUrl = await listen(candidateServer); + const proxy = createMemindCanaryProxy({ + stableTarget: stableUrl, + candidateTarget: candidateUrl, + policy: createCanaryPolicy({ usernames: ['john'] }), + identityResolver: { + async fromSessionToken() { + throw new Error('database unavailable'); + }, + async fromWechatOpenid() { + throw new Error('database unavailable'); + }, + }, + diagnosticSecret: 'diagnostic-secret', + candidateHealthTtlMs: 0, + logger: { warn() {} }, + }); + const proxyUrl = await listen(proxy.server); + t.after(async () => { + await close(proxy.server); + await close(stableServer); + await close(candidateServer); + }); + + const response = await request(proxyUrl, { + headers: { cookie: 'tkmind_user_session=john-token' }, + }); + assert.equal(response.body, 'stable:'); + assert.equal(response.headers['x-memind-runtime-route'], 'stable'); +}); + +test('candidate health requires the explicit candidate runtime role', async (t) => { + const stableServer = upstream('stable'); + const wrongRoleServer = upstream('stable'); + const stableUrl = await listen(stableServer); + const wrongRoleUrl = await listen(wrongRoleServer); + const proxy = createMemindCanaryProxy({ + stableTarget: stableUrl, + candidateTarget: wrongRoleUrl, + policy: createCanaryPolicy({ usernames: ['john'] }), + identityResolver: { + async fromSessionToken() { + return { id: 'user-john', username: 'john' }; + }, + async fromWechatOpenid() { + return null; + }, + }, + diagnosticSecret: 'diagnostic-secret', + candidateHealthTtlMs: 0, + }); + const proxyUrl = await listen(proxy.server); + t.after(async () => { + await close(proxy.server); + await close(stableServer); + await close(wrongRoleServer); + }); + + const response = await request(proxyUrl, { + headers: { cookie: 'tkmind_user_session=john-token' }, + }); + assert.equal(response.body, 'stable:'); + assert.equal(response.headers['x-memind-runtime-route'], 'stable'); +}); + +test('candidate connect race retries only replay-safe requests on stable', async (t) => { + const stableServer = upstream('stable'); + const candidateServer = candidateThatStopsAfterHealth(); + const stableUrl = await listen(stableServer); + const candidateUrl = await listen(candidateServer); + const proxy = createMemindCanaryProxy({ + stableTarget: stableUrl, + candidateTarget: candidateUrl, + policy: createCanaryPolicy({ usernames: ['john'] }), + identityResolver: { + async fromSessionToken() { + return { id: 'user-john', username: 'john' }; + }, + async fromWechatOpenid() { + return null; + }, + }, + diagnosticSecret: 'diagnostic-secret', + candidateHealthTtlMs: 0, + }); + const proxyUrl = await listen(proxy.server); + t.after(async () => { + await close(proxy.server); + await close(stableServer); + if (candidateServer.listening) await close(candidateServer); + }); + + const response = await request(proxyUrl, { + headers: { cookie: 'tkmind_user_session=john-token' }, + }); + assert.equal(response.status, 200); + assert.equal(response.body, 'stable:'); + assert.equal(response.headers['x-memind-runtime-route'], 'stable'); +}); + +test('candidate connect race never replays an unbuffered write to stable', async (t) => { + const stableServer = upstream('stable'); + const candidateServer = candidateThatStopsAfterHealth(); + const stableUrl = await listen(stableServer); + const candidateUrl = await listen(candidateServer); + const proxy = createMemindCanaryProxy({ + stableTarget: stableUrl, + candidateTarget: candidateUrl, + policy: createCanaryPolicy({ usernames: ['john'] }), + identityResolver: { + async fromSessionToken() { + return { id: 'user-john', username: 'john' }; + }, + async fromWechatOpenid() { + return null; + }, + }, + diagnosticSecret: 'diagnostic-secret', + candidateHealthTtlMs: 0, + }); + const proxyUrl = await listen(proxy.server); + t.after(async () => { + await close(proxy.server); + await close(stableServer); + if (candidateServer.listening) await close(candidateServer); + }); + + const response = await request(proxyUrl, { + method: 'POST', + path: '/api/write', + headers: { + cookie: 'tkmind_user_session=john-token', + 'content-type': 'application/json', + }, + body: '{"value":1}', + }); + assert.equal(response.status, 502); + assert.match(response.body, /canary upstream unavailable/); +}); diff --git a/package.json b/package.json index 392e4a8..d034cca 100644 --- a/package.json +++ b/package.json @@ -39,6 +39,14 @@ "build": "vite build", "build:deep-search-runtime": "node scripts/build-deep-search-runtime.mjs", "build:portal-runtime": "node scripts/build-portal-runtime.mjs", + "test:release-gate:unit": "node --test release-gate/*.test.mjs memind-canary-proxy.test.mjs server/portal-runtime-role.test.mjs", + "test:release-gate:deterministic": "node scripts/run-release-gate.mjs --mode deterministic", + "test:release-gate:scenarios": "node scripts/run-release-gate.mjs --mode scenarios", + "test:release-gate:browser": "node scripts/run-release-gate.mjs --mode browser", + "test:release-gate:providers": "node scripts/run-release-gate.mjs --mode providers", + "test:release-gate:upgrade": "node scripts/run-release-gate.mjs --mode upgrade", + "test:release-gate:all": "node scripts/run-release-gate.mjs --mode all", + "verify:release-gate-report": "node scripts/verify-release-gate-report.mjs", "build:imgproxy-runtime": "bash scripts/build-imgproxy-runtime-image.sh", "build:mindspace-service-runtime": "node scripts/build-mindspace-service-runtime.mjs", "check:mindspace-public-links": "node scripts/check-mindspace-public-links.mjs --downloads-only", diff --git a/release-gate/artifact.mjs b/release-gate/artifact.mjs index cbc81f3..9682e51 100644 --- a/release-gate/artifact.mjs +++ b/release-gate/artifact.mjs @@ -5,6 +5,7 @@ import path from 'node:path'; export const REQUIRED_PORTAL_RUNTIME_PATHS = Object.freeze([ 'server.mjs', + 'memind-canary-proxy.mjs', 'wechat-mp.bundle.mjs', 'mindspace-sandbox-mcp.mjs', 'tkmind-search-mcp.mjs', @@ -13,6 +14,9 @@ export const REQUIRED_PORTAL_RUNTIME_PATHS = Object.freeze([ 'dist', 'package.json', 'scripts/run-memind-portal-prod.sh', + 'scripts/run-memind-portal-candidate.sh', + 'scripts/run-memind-canary-proxy-prod.sh', + 'scripts/goosed-canary.compose.yml', 'scripts/check-mindspace-public-links.mjs', 'scripts/load-env.mjs', 'scripts/wechat-mp-menu.mjs', diff --git a/release-gate/coverage.mjs b/release-gate/coverage.mjs index dd59a1a..4abdb68 100644 --- a/release-gate/coverage.mjs +++ b/release-gate/coverage.mjs @@ -979,12 +979,14 @@ export const AUTOMATION_SUITES = Object.freeze([ cases: { 'REL-10': [ 'release contract: block new Agent runs, drain active work within a bound, checksum and back up, then recheck zero active runs immediately before swap', + 'canary contract: immutable user and WeChat identities route to an isolated candidate, while missing identity, resolver errors, and candidate failure fall back to stable', ], }, command: [ process.execPath, '--test', 'release-gate/canary-routing.test.mjs', + 'memind-canary-proxy.test.mjs', 'release-gate/release-runtime-safety.test.mjs', ], }, diff --git a/release-gate/release-script.test.mjs b/release-gate/release-script.test.mjs index 6f2d32f..c8f0c1c 100644 --- a/release-gate/release-script.test.mjs +++ b/release-gate/release-script.test.mjs @@ -5,6 +5,7 @@ import path from 'node:path'; import test from 'node:test'; const ROOT = path.resolve(new URL('..', import.meta.url).pathname); +const CANARY_RELEASE = path.join(ROOT, 'scripts', 'release-portal-canary-prod.sh'); test('production release verifies gate report before any 103 connection', async () => { const source = await fs.readFile( @@ -77,3 +78,51 @@ test('production release rejects --skip-tests before repository or network prefl assert.match(result.stderr, /禁止 --skip-tests/); assert.doesNotMatch(`${result.stdout}\n${result.stderr}`, /103 只读预检/); }); + +test('production canary verifies the exact Gate artifact before any 103 preflight or upload', async () => { + const source = await fs.readFile(CANARY_RELEASE, 'utf8'); + const gateIndex = source.indexOf('verify-release-gate-report.mjs'); + const preflightIndex = source.indexOf('Run 103 read-only preflight'); + const uploadIndex = source.indexOf('Upload the verified candidate bundle'); + assert.ok(gateIndex > 0, 'missing canary Gate verifier'); + assert.ok(preflightIndex > gateIndex, '103 preflight must follow Gate verification'); + assert.ok(uploadIndex > preflightIndex, 'upload must follow read-only preflight'); + assert.match(source, /branch.*!= "main"/); + assert.match(source, /rev-parse origin\/main/); + assert.match(source, /Production canary release forbids/); +}); + +test('production canary keeps stable 8081 live and switches only after verified backups and fallback', async () => { + const source = await fs.readFile(CANARY_RELEASE, 'utf8'); + const fullBackup = source.indexOf('Create and verify the full stable backup'); + const persistBackup = source.indexOf('Create and verify the persisted-data backup'); + const goosedStart = source.indexOf('Start an isolated goosed candidate on 18015'); + const candidateStart = source.indexOf('Start the passive candidate Portal on 18081'); + const proxyStart = source.indexOf('Start the fail-closed identity router on 18080'); + const tunnelSwitch = source.indexOf('Switch only the reverse tunnel'); + const fallbackProbe = source.indexOf('route-probe?username=john'); + assert.ok(fullBackup > 0); + assert.ok(persistBackup > fullBackup); + assert.ok(goosedStart > persistBackup); + assert.ok(candidateStart > goosedStart); + assert.ok(proxyStart > candidateStart); + assert.ok(tunnelSwitch > proxyStart); + assert.ok(fallbackProbe > tunnelSwitch); + assert.match(source, /write_tunnel_plist 8081/); + assert.match(source, /CANARY_USERNAMES="john"/); + assert.match(source, /CANARY_WECHAT_USER_IDS="wx_ul610et8"/); + assert.match(source, /route-probe\?\$\{probe_query\}/); + assert.match(source, /candidate_healthy/); + assert.match(source, /MEMIND_CANARY_RELEASE_ID/); + assert.doesNotMatch(source, /lsof -tiTCP:8081/); + assert.doesNotMatch(source, /bootout.*cn\.tkmind\.memind-portal/); +}); + +test('release readiness ignores only the generated canary artifact cover path', async () => { + const source = await fs.readFile( + path.join(ROOT, 'scripts', 'check-release-ready.sh'), + 'utf8', + ); + assert.match(source, /:\(exclude\)\.runtime\/portal\/public\/plaza-covers\/\*\*/); + assert.doesNotMatch(source, /:\(exclude\)\.runtime\/\*\*/); +}); diff --git a/scripts/build-portal-runtime.mjs b/scripts/build-portal-runtime.mjs index 8d550e3..aef8117 100755 --- a/scripts/build-portal-runtime.mjs +++ b/scripts/build-portal-runtime.mjs @@ -136,6 +136,26 @@ async function bundleServer() { await run(esbuildBin, args); } +async function bundleCanaryProxy() { + if (!(await exists(esbuildBin))) { + throw new Error(`未找到 esbuild: ${esbuildBin}`); + } + console.log('==> 打包用户级灰度路由器'); + const args = [ + 'memind-canary-proxy.mjs', + '--bundle', + '--platform=node', + '--format=esm', + `--target=${runtimeNodeTarget}`, + '--outfile=.runtime/portal/memind-canary-proxy.mjs', + '--banner:js=import { createRequire as __createRequire } from "node:module"; const require = __createRequire(import.meta.url);', + ]; + for (const pkg of externalPackages) { + args.push(`--external:${pkg}`); + } + await run(esbuildBin, args); +} + async function bundleSandboxMcp() { if (!(await exists(esbuildBin))) { throw new Error(`未找到 esbuild: ${esbuildBin}`); @@ -386,6 +406,21 @@ async function writeMetadata() { } else { throw new Error(`缺少 Portal 启动脚本: ${prodStartScript}`); } + for (const scriptName of [ + 'run-memind-portal-candidate.sh', + 'run-memind-canary-proxy-prod.sh', + ]) { + const source = path.join(root, 'scripts', scriptName); + if (!(await exists(source))) { + throw new Error(`缺少灰度运行脚本: ${source}`); + } + await fs.copyFile(source, path.join(runtimeScriptsDir, scriptName)); + await fs.chmod(path.join(runtimeScriptsDir, scriptName), 0o755); + } + await fs.copyFile( + path.join(root, 'scripts', 'goosed-canary.compose.yml'), + path.join(runtimeScriptsDir, 'goosed-canary.compose.yml'), + ); await fs.copyFile( path.join(root, 'scripts', 'load-env.mjs'), @@ -491,6 +526,7 @@ async function writeMetadata() { ' tkmind-search-mcp.mjs (esbuild bundle; required when MindSearch is enabled)', ' tkmind-excel-mcp.mjs (esbuild bundle; required when Excel Analyst is enabled)', ' wechat-mp.bundle.mjs (esbuild bundle; hot-swappable WeChat MP module)', + ' memind-canary-proxy.mjs (immutable-identity stable/candidate router)', '', 'Post-deploy validation (scripts/check-mindspace-public-links.mjs):', ' Scans MindSpace/*/public/*.html for missing relative href/src/cover targets.', @@ -516,6 +552,7 @@ async function writeMetadata() { ' Current public path: m.tkmind.cn -> 105 nginx -> 103 Portal :8081', ' No separate public test domain remains; use explicit local/test overrides only when needed.', ' Production H5 and WeChat links should use m.tkmind.cn unless a test rollout explicitly overrides H5_PUBLIC_BASE_URL.', + ' Canary layout: stable :8081, candidate :18081, identity router :18080, candidate goosed :18015.', '', 'Streaming runtime operations:', ' node scripts/check-stream-runtime.mjs', @@ -560,6 +597,7 @@ async function main() { await buildFrontend(); await copyRuntimeAssets(); await bundleServer(); + await bundleCanaryProxy(); await bundleWechatMp(); await bundleSandboxMcp(); await bundleMindSearchMcp(); @@ -571,6 +609,8 @@ async function main() { await copyMindspacePublicLinkTools(); await writeMetadata(); await fs.chmod(path.join(runtimeRoot, 'scripts', 'run-memind-portal-prod.sh'), 0o755); + await fs.chmod(path.join(runtimeRoot, 'scripts', 'run-memind-portal-candidate.sh'), 0o755); + await fs.chmod(path.join(runtimeRoot, 'scripts', 'run-memind-canary-proxy-prod.sh'), 0o755); await fs.chmod(path.join(runtimeRoot, 'scripts', 'wechat-mp-menu.mjs'), 0o755); await fs.chmod(path.join(runtimeRoot, 'scripts', 'check-stream-runtime.mjs'), 0o755); await fs.chmod(path.join(runtimeRoot, 'scripts', 'runtime-worker-drain.mjs'), 0o755); diff --git a/scripts/check-release-ready.sh b/scripts/check-release-ready.sh index 00ccd8e..dcfd4e5 100755 --- a/scripts/check-release-ready.sh +++ b/scripts/check-release-ready.sh @@ -53,9 +53,16 @@ case "${branch}" in ;; esac -if [[ -n "$(git -C "${ROOT}" status --porcelain=v1 --untracked-files=all)" ]]; then +# The runtime builder intentionally strips this legacy tracked content from the +# source-free artifact. The complete Gate applies the same narrow exception. +relevant_status="$( + git -C "${ROOT}" status --porcelain=v1 --untracked-files=all -- \ + . \ + ':(exclude).runtime/portal/public/plaza-covers/**' +)" +if [[ -n "${relevant_status}" ]]; then echo "Release check failed: worktree has uncommitted or untracked changes." >&2 - git -C "${ROOT}" status --short --untracked-files=all >&2 + printf '%s\n' "${relevant_status}" >&2 exit 1 fi diff --git a/scripts/goosed-canary.compose.yml b/scripts/goosed-canary.compose.yml new file mode 100644 index 0000000..93e6266 --- /dev/null +++ b/scripts/goosed-canary.compose.yml @@ -0,0 +1,39 @@ +services: + goosed-canary: + image: tkmind/goosed:prod-${GOOSED_TAG:-latest} + container_name: goosed-prod-canary + restart: unless-stopped + env_file: ${GOOSED_PROD_ENV_FILE:-/Users/john/Project/goosed-prod/.env} + environment: + GOOSE_PORT: 18006 + GOOSE_AIDER_BIN: /usr/local/bin/aider + GOOSE_OPENHANDS_BIN: /usr/local/bin/openhands + GOOSE_OPENHANDS_RUNNER: host + GOOSE_OPENHANDS_TIMEOUT_SECS: "900" + ports: + - "18015:18006" + volumes: + - ${CANDIDATE_RUNTIME_DIR:?}:/opt/portal:ro + - ${GOOSED_RUNTIME_TOOLS_DIR:?}:/opt/tools:ro + - ${GOOSED_RUNTIME_TOOLS_DIR:?}/bin/aider:/usr/local/bin/aider:ro + - ${GOOSED_RUNTIME_TOOLS_DIR:?}/bin/openhands:/usr/local/bin/openhands:ro + - ${GOOSED_RUNTIME_TOOLS_DIR:?}/openhands-data:/opt/tools/openhands-data + - ${GOOSED_HARNESS_DIR:?}:/home/goose/.codex/harness + - ${MINDSPACE_ROOT:?}:${MINDSPACE_ROOT:?} + - ${MINDSPACE_STORAGE_ROOT:?}:${MINDSPACE_STORAGE_ROOT:?} + - goosed-canary-data:/home/goose/.local/share/goose + healthcheck: + test: ["CMD", "curl", "-kfsS", "https://127.0.0.1:18006/status"] + interval: 15s + timeout: 5s + retries: 3 + start_period: 20s + networks: + - goosed-net + +volumes: + goosed-canary-data: + +networks: + goosed-net: + driver: bridge diff --git a/scripts/release-portal-canary-prod.sh b/scripts/release-portal-canary-prod.sh new file mode 100755 index 0000000..6b16170 --- /dev/null +++ b/scripts/release-portal-canary-prod.sh @@ -0,0 +1,575 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +HOST="${STUDIO_HOST:-58.38.22.103}" +REMOTE_ROOT="${STUDIO_REMOTE_ROOT:-/Users/john/Project}" +STABLE_DIR="${REMOTE_ROOT}/Memind" +CANDIDATE_BASE="${REMOTE_ROOT}/Memind-candidates" +INCOMING_DIR="${REMOTE_ROOT}/incoming/memind-portal-canary" +BACKUP_DIR="${REMOTE_ROOT}/backups/memind" +GOOSED_DIR="${REMOTE_ROOT}/goosed-prod" +RUNTIME_ROOT="${ROOT}/.runtime/portal" +PORTAL_CANDIDATE_LABEL="cn.tkmind.memind-portal-candidate" +CANARY_PROXY_LABEL="cn.tkmind.memind-canary-proxy" +PORTAL_TUNNEL_LABEL="cn.tkmind.memind-portal-tunnel" +MEMIND_PORTAL_TUNNEL_HOST="${MEMIND_PORTAL_TUNNEL_HOST:-ssh105-public}" +MEMIND_PORTAL_TUNNEL_REMOTE_PORT="${MEMIND_PORTAL_TUNNEL_REMOTE_PORT:-19081}" +CANARY_USERNAMES="john" +CANARY_WECHAT_USER_IDS="wx_ul610et8" +RELEASE_TS="$(date +%Y%m%d-%H%M%S)" +FULL_SHA="$(git -C "${ROOT}" rev-parse HEAD)" +SHORT_SHA="${FULL_SHA:0:7}" +RELEASE_ID="${RELEASE_TS}-${SHORT_SHA}" +TMP_DIR="$(mktemp -d "${TMPDIR:-/tmp}/memind-portal-canary-release.XXXXXX")" +BUNDLE_PATH="${TMP_DIR}/memind-portal-canary-${RELEASE_ID}.tar.gz" +MANIFEST_PATH="${TMP_DIR}/memind-portal-canary-${RELEASE_ID}.manifest.txt" +SHA_PATH="${TMP_DIR}/memind-portal-canary-${RELEASE_ID}.sha256" +DRY_RUN=0 +AUTO_YES=0 + +cleanup() { + rm -rf "${TMP_DIR}" +} +trap cleanup EXIT + +usage() { + cat <<'EOF' +Usage: + bash scripts/release-portal-canary-prod.sh [--dry-run] [--yes] + +Deploys a candidate Portal beside the stable 103 runtime: + stable Portal 127.0.0.1:8081 + canary router 127.0.0.1:18080 + candidate Portal 127.0.0.1:18081 + candidate goosed 127.0.0.1:18015 + +The stable runtime is not replaced. The 105 reverse tunnel moves to the canary +router only after backups, candidate health, immutable identity checks, and +automatic fallback checks pass. +EOF +} + +while [[ $# -gt 0 ]]; do + case "$1" in + --dry-run) DRY_RUN=1 ;; + --yes|-y) AUTO_YES=1 ;; + -h|--help) + usage + exit 0 + ;; + *) + echo "Unknown argument: $1" >&2 + usage >&2 + exit 1 + ;; + esac + shift +done + +for bypass in \ + ALLOW_PORTAL_RELEASE_SCOPE_BYPASS \ + ALLOW_MINDSPACE_PUBLIC_LINK_ISSUES \ + MEMIND_RELEASE_SKIP_GATE; do + if [[ "${!bypass:-0}" == "1" ]]; then + echo "Production canary release forbids ${bypass}." >&2 + exit 1 + fi +done + +say() { + printf '\n[%s] %s\n' "$(date +%H:%M:%S)" "$*" +} + +need_cmd() { + command -v "$1" >/dev/null 2>&1 || { + echo "Missing command: $1" >&2 + exit 1 + } +} + +need_cmd ssh +need_cmd scp +need_cmd tar +need_cmd shasum + +branch="$(git -C "${ROOT}" branch --show-current)" +if [[ "${branch}" != "main" ]]; then + echo "Canary production release requires main, got: ${branch:-detached}" >&2 + exit 1 +fi +git -C "${ROOT}" fetch origin --prune +if [[ "$(git -C "${ROOT}" rev-parse HEAD)" != "$(git -C "${ROOT}" rev-parse origin/main)" ]]; then + echo "Canary production release requires HEAD to equal origin/main." >&2 + exit 1 +fi +ALLOW_MAIN_RELEASE=1 bash "${ROOT}/scripts/check-release-ready.sh" --skip-fetch + +say "Run release source guards" +( + cd "${ROOT}" + npm test -- --test-name-pattern='publish|space|billing' >/dev/null + npm run verify:mindspace-publish-guards >/dev/null + npm run verify:mindspace-page-sync-guards >/dev/null + npm run verify:h5-session-patches >/dev/null + npm run verify:page-data >/dev/null + npm run check:mindspace-public-links >/dev/null +) + +required_runtime_paths=( + server.mjs + memind-canary-proxy.mjs + wechat-mp.bundle.mjs + mindspace-sandbox-mcp.mjs + tkmind-search-mcp.mjs + tkmind-excel-mcp.mjs + dist + package.json + scripts/run-memind-portal-candidate.sh + scripts/run-memind-canary-proxy-prod.sh + scripts/goosed-canary.compose.yml + scripts/memind-portal-tunnel.sh +) +for relative in "${required_runtime_paths[@]}"; do + [[ -e "${RUNTIME_ROOT}/${relative}" ]] || { + echo "Candidate runtime is missing ${relative}; rebuild and rerun the complete Gate." >&2 + exit 1 + } +done + +say "Verify the complete Gate report against the exact candidate artifact" +node "${ROOT}/scripts/verify-release-gate-report.mjs" --artifact "${RUNTIME_ROOT}" + +say "Package the verified candidate artifact" +tar -czf "${BUNDLE_PATH}" -C "${RUNTIME_ROOT}" . +bundle_sha="$(shasum -a 256 "${BUNDLE_PATH}" | awk '{print $1}')" +printf '%s %s\n' "${bundle_sha}" "$(basename "${BUNDLE_PATH}")" > "${SHA_PATH}" +{ + echo "release_id=${RELEASE_ID}" + echo "created_at=$(date '+%Y-%m-%d %H:%M:%S %z')" + echo "git_head=${FULL_SHA}" + echo "git_branch=${branch}" + echo "artifact_tree=.runtime/portal" + echo "artifact_bundle_sha256=${bundle_sha}" + echo "routing=stable:8081,proxy:18080,candidate:18081,goosed-canary:18015" + echo "canary_usernames=${CANARY_USERNAMES}" + echo "canary_wechat_user_ids=${CANARY_WECHAT_USER_IDS}" + echo "manual_environment_changes=none" +} > "${MANIFEST_PATH}" + +if [[ "${DRY_RUN}" -eq 1 ]]; then + say "Dry-run complete; no 103 connection or production write occurred" + ls -lh "${BUNDLE_PATH}" "${MANIFEST_PATH}" "${SHA_PATH}" + exit 0 +fi + +say "Run 103 read-only preflight" +ssh -o BatchMode=yes -o ConnectTimeout=15 "${HOST}" \ + "test -d '${STABLE_DIR}' \ + && test -f '${STABLE_DIR}/.env' \ + && test -f '${GOOSED_DIR}/docker-compose.prod.yml' \ + && curl -fsS http://127.0.0.1:8081/api/status >/dev/null \ + && /opt/homebrew/bin/docker inspect goosed-prod-1 >/dev/null \ + && test \"\$(df -Pk '${REMOTE_ROOT}' | awk 'NR==2 {print \$4}')\" -gt 10485760" + +if [[ "${AUTO_YES}" -ne 1 ]]; then + say "Production canary confirmation" + echo "Target host: ${HOST}" + echo "Release: ${RELEASE_ID}" + echo "Commit: ${FULL_SHA}" + echo "Stable remains on 8081; selected immutable identities route to candidate 18081." + read -r -p "Continue with the 103 canary release? [y/N] " confirm /dev/null || true + )" + if [[ "${code}" == "200" ]] \ + && grep -qi '^x-memind-runtime-role: candidate' "${headers}"; then + rm -f "${headers}" + return 0 + fi + rm -f "${headers}" + return 1 +} + +write_tunnel_plist() { + local local_port="$1" + cat > "${TUNNEL_PLIST}" < + + + + Label${PORTAL_TUNNEL_LABEL} + ProgramArguments + ${STABLE_DIR}/scripts/memind-portal-tunnel.sh + EnvironmentVariables + + MEMIND_PORTAL_TUNNEL_HOST${MEMIND_PORTAL_TUNNEL_HOST} + MEMIND_PORTAL_TUNNEL_LOCAL_PORT${local_port} + MEMIND_PORTAL_TUNNEL_REMOTE_PORT${MEMIND_PORTAL_TUNNEL_REMOTE_PORT} + + RunAtLoad + KeepAlive + StandardOutPath${HOME}/Library/Logs/memind-portal-tunnel.log + StandardErrorPath${HOME}/Library/Logs/memind-portal-tunnel.log + + +EOF + launchctl bootout "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null 2>&1 || true + launchctl bootstrap "${LAUNCHD_GUI}" "${TUNNEL_PLIST}" >/dev/null + launchctl enable "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null 2>&1 || true + launchctl kickstart -k "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null +} + +stop_candidate_services() { + launchctl bootout "${LAUNCHD_GUI}/${CANARY_PROXY_LABEL}" >/dev/null 2>&1 || true + launchctl bootout "${LAUNCHD_GUI}/${PORTAL_CANDIDATE_LABEL}" >/dev/null 2>&1 || true + "${DOCKER_BIN}" rm -f goosed-prod-canary >/dev/null 2>&1 || true +} + +rollback() { + local status=$? + trap - ERR + say "Canary rollout failed; restoring stable-only tunnel" + rm -f "${STABLE_DIR}/.release-drain" + write_tunnel_plist 8081 >/dev/null 2>&1 || true + stop_candidate_services + exit "${status}" +} +trap rollback ERR + +[[ -f "${BUNDLE}" && -f "${MANIFEST}" && -f "${SHA_FILE}" ]] +cd "${INCOMING_DIR}" +shasum -a 256 -c "$(basename "${SHA_FILE}")" + +say "Create and verify the full stable backup" +COPYFILE_DISABLE=1 tar --exclude='Memind/.tailscale/*.sock' \ + -czf "${FULL_BACKUP_TAR}" -C "$(dirname "${STABLE_DIR}")" "$(basename "${STABLE_DIR}")" +shasum -a 256 "${FULL_BACKUP_TAR}" > "${FULL_BACKUP_TAR}.sha256" +shasum -a 256 -c "${FULL_BACKUP_TAR}.sha256" +gzip -t "${FULL_BACKUP_TAR}" +tar -tzf "${FULL_BACKUP_TAR}" | grep '^Memind/.env$' >/dev/null + +say "Create and verify the persisted-data backup" +persist_tmp="$(mktemp -d "${TMPDIR:-/tmp}/memind-canary-persist.XXXXXX")" +for item in .env MindSpace data users .tailscale public/plaza-covers logs; do + if [[ -e "${STABLE_DIR}/${item}" ]]; then + mkdir -p "${persist_tmp}/$(dirname "${item}")" + if [[ "${item}" == ".tailscale" ]]; then + mkdir -p "${persist_tmp}/${item}" + ( + cd "${STABLE_DIR}/${item}" + COPYFILE_DISABLE=1 tar --exclude='*.sock' -cf - . + ) | ( + cd "${persist_tmp}/${item}" + COPYFILE_DISABLE=1 tar -xf - + ) + else + cp -a "${STABLE_DIR}/${item}" "${persist_tmp}/$(dirname "${item}")/" + fi + fi +done +COPYFILE_DISABLE=1 tar -czf "${PERSIST_BACKUP_TAR}" -C "${persist_tmp}" . +rm -rf "${persist_tmp}" +shasum -a 256 "${PERSIST_BACKUP_TAR}" > "${PERSIST_BACKUP_TAR}.sha256" +shasum -a 256 -c "${PERSIST_BACKUP_TAR}.sha256" +gzip -t "${PERSIST_BACKUP_TAR}" +tar -tzf "${PERSIST_BACKUP_TAR}" | grep -E '^(\./)?\.env$' >/dev/null + +if [[ -f "${TUNNEL_PLIST}" ]]; then + cp "${TUNNEL_PLIST}" "${TUNNEL_PLIST_BACKUP}" +fi + +say "Extract the source-free candidate runtime" +rm -rf "${CANDIDATE_DIR}" +mkdir -p "${CANDIDATE_DIR}" +tar -xzf "${BUNDLE}" -C "${CANDIDATE_DIR}" +cp "${MANIFEST}" "${CANDIDATE_DIR}/.release-manifest.txt" +chmod 755 \ + "${CANDIDATE_DIR}/scripts/run-memind-portal-candidate.sh" \ + "${CANDIDATE_DIR}/scripts/run-memind-canary-proxy-prod.sh" + +say "Stop an older canary without touching stable Portal 8081" +if curl -fsS http://127.0.0.1:18080/api/status >/dev/null 2>&1; then + write_tunnel_plist 8081 + sleep 2 +fi +stop_candidate_services + +say "Start an isolated goosed candidate on 18015" +( + cd "${GOOSED_DIR}" + set -a + # shellcheck disable=SC1091 + source .env + set +a + export CANDIDATE_RUNTIME_DIR="${CANDIDATE_DIR}" + "${DOCKER_BIN}" compose -p goosed-prod \ + -f "${GOOSED_COMPOSE}" \ + -f "${CANDIDATE_DIR}/scripts/goosed-canary.compose.yml" \ + up -d --no-deps goosed-canary +) +for _ in $(seq 1 60); do + if [[ "$(curl -skS -m 5 https://127.0.0.1:18015/status 2>/dev/null || true)" == "ok" ]]; then + break + fi + sleep 2 +done +[[ "$(curl -skS -m 5 https://127.0.0.1:18015/status 2>/dev/null || true)" == "ok" ]] +"${DOCKER_BIN}" exec goosed-prod-canary \ + sh -lc 'test -x /usr/local/bin/node && test -f /opt/portal/mindspace-sandbox-mcp.mjs' + +say "Start the passive candidate Portal on 18081" +cat > "${CANDIDATE_PLIST}" < + + + + Label${PORTAL_CANDIDATE_LABEL} + ProgramArguments + ${CANDIDATE_DIR}/scripts/run-memind-portal-candidate.sh + EnvironmentVariables + + MEMIND_CANARY_STABLE_ROOT${STABLE_DIR} + MEMIND_CANARY_RELEASE_ID${RELEASE_ID} + MEMIND_CANARY_CANDIDATE_PORT18081 + MEMIND_CANARY_GOOSED_URLhttps://127.0.0.1:18015 + + RunAtLoad + KeepAlive + ThrottleInterval5 + StandardOutPath${HOME}/Library/Logs/memind-portal-candidate.log + StandardErrorPath${HOME}/Library/Logs/memind-portal-candidate.log + + +EOF +launchctl bootstrap "${LAUNCHD_GUI}" "${CANDIDATE_PLIST}" >/dev/null +launchctl kickstart -k "${LAUNCHD_GUI}/${PORTAL_CANDIDATE_LABEL}" >/dev/null +for _ in $(seq 1 60); do + candidate_healthy && break + sleep 2 +done +candidate_healthy + +say "Validate immutable canary selectors against the production identity database" +mkdir -p "$(dirname "${SECRET_FILE}")" +if [[ ! -s "${SECRET_FILE}" ]]; then + umask 077 + /usr/bin/openssl rand -hex 32 > "${SECRET_FILE}" +fi +chmod 600 "${SECRET_FILE}" +( + set -a + # shellcheck disable=SC1091 + source "${STABLE_DIR}/.env" + set +a + export MEMIND_PORTAL_H5_ROOT="${STABLE_DIR}" + export MEMIND_RELEASE_CANARY_USERNAMES="${CANARY_USERNAMES}" + export MEMIND_RELEASE_CANARY_WECHAT_USER_IDS="${CANARY_WECHAT_USER_IDS}" + /opt/homebrew/opt/node@24/bin/node "${CANDIDATE_DIR}/memind-canary-proxy.mjs" --check +) + +say "Start the fail-closed identity router on 18080" +cat > "${PROXY_PLIST}" < + + + + Label${CANARY_PROXY_LABEL} + ProgramArguments + ${CANDIDATE_DIR}/scripts/run-memind-canary-proxy-prod.sh + EnvironmentVariables + + MEMIND_CANARY_STABLE_ROOT${STABLE_DIR} + MEMIND_CANARY_SECRET_FILE${SECRET_FILE} + MEMIND_RELEASE_CANARY_USERNAMES${CANARY_USERNAMES} + MEMIND_RELEASE_CANARY_WECHAT_USER_IDS${CANARY_WECHAT_USER_IDS} + + RunAtLoad + KeepAlive + ThrottleInterval5 + StandardOutPath${HOME}/Library/Logs/memind-canary-proxy.log + StandardErrorPath${HOME}/Library/Logs/memind-canary-proxy.log + + +EOF +launchctl bootstrap "${LAUNCHD_GUI}" "${PROXY_PLIST}" >/dev/null +launchctl kickstart -k "${LAUNCHD_GUI}/${CANARY_PROXY_LABEL}" >/dev/null +sleep 2 +diagnostic_secret="$(tr -d '\r\n' < "${SECRET_FILE}")" +diagnostic="$( + curl -fsS \ + -H "X-Memind-Canary-Secret: ${diagnostic_secret}" \ + http://127.0.0.1:18080/__memind_canary/health +)" +printf '%s' "${diagnostic}" | /opt/homebrew/opt/node@24/bin/node --input-type=module -e ' + let input = ""; + process.stdin.on("data", (chunk) => { input += chunk; }); + process.stdin.on("end", () => { + const status = JSON.parse(input); + if (!status.ok || !status.stable || !status.candidate) process.exit(1); + if (status.policy.usernames < 1 || status.policy.wechatUserIds < 1) process.exit(1); + }); +' +for probe_query in 'username=john' 'wechat_user_id=wx_ul610et8'; do + route_probe="$( + curl -fsS \ + -H "X-Memind-Canary-Secret: ${diagnostic_secret}" \ + "http://127.0.0.1:18080/__memind_canary/route-probe?${probe_query}" + )" + printf '%s' "${route_probe}" | /opt/homebrew/opt/node@24/bin/node --input-type=module -e ' + let input = ""; + process.stdin.on("data", (chunk) => { input += chunk; }); + process.stdin.on("end", () => { + const status = JSON.parse(input); + if (status.configuredTarget !== "candidate") process.exit(1); + if (status.candidate !== true || status.route !== "candidate") process.exit(1); + }); + ' +done + +say "Drain active Agent runs for the bounded tunnel switch" +touch "${STABLE_DIR}/.release-drain" +deadline=$(( $(date +%s) + 120 )) +while (( $(date +%s) < deadline )); do + queue="$( + /opt/homebrew/opt/node@24/bin/node \ + "${STABLE_DIR}/scripts/agent-run-worker.mjs" --status 2>/dev/null || true + )" + counts="$( + printf '%s' "${queue}" | /opt/homebrew/opt/node@24/bin/node --input-type=module -e ' + let input = ""; + process.stdin.on("data", (chunk) => { input += chunk; }); + process.stdin.on("end", () => { + try { + const queue = JSON.parse(input).queue || {}; + console.log(`${Number(queue.inFlight || 0)} ${Number(queue.pendingDispatches || 0)}`); + } catch { + console.log("unknown unknown"); + } + }); + ' + )" + if [[ "${counts}" == "0 0" ]]; then + break + fi + sleep 2 +done +[[ "${counts:-unknown unknown}" == "0 0" ]] + +say "Switch only the reverse tunnel from stable 8081 to identity router 18080" +write_tunnel_plist 18080 +sleep 2 +remote_code="$( + ssh -o BatchMode=yes -o ConnectTimeout=10 "${MEMIND_PORTAL_TUNNEL_HOST}" \ + "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}/api/status" \ + 2>/dev/null || true +)" +[[ "${remote_code}" == "200" ]] +rm -f "${STABLE_DIR}/.release-drain" + +say "Verify candidate failure automatically falls back to stable" +launchctl bootout "${LAUNCHD_GUI}/${PORTAL_CANDIDATE_LABEL}" >/dev/null +for _ in $(seq 1 20); do + if ! curl -fsS http://127.0.0.1:18081/api/status >/dev/null 2>&1; then + break + fi + sleep 1 +done +fallback_probe="$( + curl -fsS \ + -H "X-Memind-Canary-Secret: ${diagnostic_secret}" \ + 'http://127.0.0.1:18080/__memind_canary/route-probe?username=john' +)" +printf '%s' "${fallback_probe}" | /opt/homebrew/opt/node@24/bin/node --input-type=module -e ' + let input = ""; + process.stdin.on("data", (chunk) => { input += chunk; }); + process.stdin.on("end", () => { + const status = JSON.parse(input); + if (status.configuredTarget !== "candidate") process.exit(1); + if (status.candidate !== false || status.route !== "stable") process.exit(1); + }); +' +launchctl bootstrap "${LAUNCHD_GUI}" "${CANDIDATE_PLIST}" >/dev/null +launchctl kickstart -k "${LAUNCHD_GUI}/${PORTAL_CANDIDATE_LABEL}" >/dev/null +for _ in $(seq 1 60); do + candidate_healthy && break + sleep 2 +done +candidate_healthy +final_diagnostic="$( + curl -fsS \ + -H "X-Memind-Canary-Secret: ${diagnostic_secret}" \ + http://127.0.0.1:18080/__memind_canary/health +)" +printf '%s' "${final_diagnostic}" | /opt/homebrew/opt/node@24/bin/node --input-type=module -e ' + let input = ""; + process.stdin.on("data", (chunk) => { input += chunk; }); + process.stdin.on("end", () => { + const status = JSON.parse(input); + if (!status.ok || !status.stable || !status.candidate) process.exit(1); + }); +' + +trap - ERR +say "103 production canary is active" +printf 'release_id=%s\n' "${RELEASE_ID}" +printf 'candidate_dir=%s\n' "${CANDIDATE_DIR}" +printf 'full_backup=%s\n' "${FULL_BACKUP_TAR}" +printf 'persist_backup=%s\n' "${PERSIST_BACKUP_TAR}" +printf 'stable_health=http://127.0.0.1:8081/api/status\n' +printf 'proxy_health=http://127.0.0.1:18080/__memind_canary/health\n' +printf 'candidate_health=http://127.0.0.1:18081/api/status\n' +printf 'candidate_goosed=https://127.0.0.1:18015/status\n' +REMOTE_SCRIPT + +say "103 canary release completed" +printf 'release_id=%s\n' "${RELEASE_ID}" +printf 'git_head=%s\n' "${FULL_SHA}" diff --git a/scripts/release-portal-runtime-prod.sh b/scripts/release-portal-runtime-prod.sh index bd14de4..c97ff02 100755 --- a/scripts/release-portal-runtime-prod.sh +++ b/scripts/release-portal-runtime-prod.sh @@ -84,8 +84,9 @@ if [[ "${ALLOW_MINDSPACE_PUBLIC_LINK_ISSUES:-0}" == "1" ]]; then fi if [[ "${DRY_RUN}" -eq 0 ]]; then - echo "当前脚本执行整包替换,不是用户级灰度入口;在候选 runtime + 灰度代理上线前禁止生产发布。" >&2 - echo "请先使用 docs/release-canary-103.md 规定的灰度流程,并由守门员重新授权。" >&2 + echo "当前脚本执行 8081 整包晋升,不是用户级灰度入口。" >&2 + echo "在同一候选完成 103 灰度验收且晋升证据校验落地前,禁止非 dry-run。" >&2 + echo "首次生产动作请使用 scripts/release-portal-canary-prod.sh。" >&2 exit 1 fi @@ -165,7 +166,7 @@ fi verify_runtime_artifact() { local missing=0 - for required in server.mjs wechat-mp.bundle.mjs mindspace-sandbox-mcp.mjs tkmind-search-mcp.mjs tkmind-excel-mcp.mjs mindspace-public-links.mjs dist package.json scripts/run-memind-portal-prod.sh scripts/check-mindspace-public-links.mjs scripts/load-env.mjs scripts/wechat-mp-menu.mjs scripts/memind-portal-tunnel.sh; do + for required in server.mjs memind-canary-proxy.mjs wechat-mp.bundle.mjs mindspace-sandbox-mcp.mjs tkmind-search-mcp.mjs tkmind-excel-mcp.mjs mindspace-public-links.mjs dist package.json scripts/run-memind-portal-prod.sh scripts/run-memind-portal-candidate.sh scripts/run-memind-canary-proxy-prod.sh scripts/goosed-canary.compose.yml scripts/check-mindspace-public-links.mjs scripts/load-env.mjs scripts/wechat-mp-menu.mjs scripts/memind-portal-tunnel.sh; do if [[ ! -e "${RUNTIME_ROOT}/${required}" ]]; then echo "runtime 产物缺失: ${RUNTIME_ROOT}/${required}" >&2 missing=1 diff --git a/scripts/rollback-portal-canary-prod.sh b/scripts/rollback-portal-canary-prod.sh new file mode 100755 index 0000000..6f8d6ab --- /dev/null +++ b/scripts/rollback-portal-canary-prod.sh @@ -0,0 +1,81 @@ +#!/usr/bin/env bash +set -euo pipefail + +HOST="${STUDIO_HOST:-58.38.22.103}" +STABLE_DIR="${STUDIO_REMOTE_ROOT:-/Users/john/Project}/Memind" +PORTAL_CANDIDATE_LABEL="cn.tkmind.memind-portal-candidate" +CANARY_PROXY_LABEL="cn.tkmind.memind-canary-proxy" +PORTAL_TUNNEL_LABEL="cn.tkmind.memind-portal-tunnel" +MEMIND_PORTAL_TUNNEL_HOST="${MEMIND_PORTAL_TUNNEL_HOST:-ssh105-public}" +MEMIND_PORTAL_TUNNEL_REMOTE_PORT="${MEMIND_PORTAL_TUNNEL_REMOTE_PORT:-19081}" +AUTO_YES=0 + +if [[ "${1:-}" == "--yes" || "${1:-}" == "-y" ]]; then + AUTO_YES=1 +elif [[ $# -gt 0 ]]; then + echo "Usage: bash scripts/rollback-portal-canary-prod.sh [--yes]" >&2 + exit 1 +fi + +if [[ "${AUTO_YES}" -ne 1 ]]; then + echo "This restores the 105 tunnel to stable Portal 8081 and stops only canary services." + read -r -p "Continue with 103 canary rollback? [y/N] " confirm "${TUNNEL_PLIST}" < + + + + Label${PORTAL_TUNNEL_LABEL} + ProgramArguments + ${STABLE_DIR}/scripts/memind-portal-tunnel.sh + EnvironmentVariables + + MEMIND_PORTAL_TUNNEL_HOST${MEMIND_PORTAL_TUNNEL_HOST} + MEMIND_PORTAL_TUNNEL_LOCAL_PORT8081 + MEMIND_PORTAL_TUNNEL_REMOTE_PORT${MEMIND_PORTAL_TUNNEL_REMOTE_PORT} + + RunAtLoad + KeepAlive + StandardOutPath${HOME}/Library/Logs/memind-portal-tunnel.log + StandardErrorPath${HOME}/Library/Logs/memind-portal-tunnel.log + + +EOF + +launchctl bootout "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null 2>&1 || true +launchctl bootstrap "${LAUNCHD_GUI}" "${TUNNEL_PLIST}" >/dev/null +launchctl kickstart -k "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null +launchctl bootout "${LAUNCHD_GUI}/${CANARY_PROXY_LABEL}" >/dev/null 2>&1 || true +launchctl bootout "${LAUNCHD_GUI}/${PORTAL_CANDIDATE_LABEL}" >/dev/null 2>&1 || true +/opt/homebrew/bin/docker rm -f goosed-prod-canary >/dev/null 2>&1 || true +rm -f "${STABLE_DIR}/.release-drain" + +curl -fsS http://127.0.0.1:8081/api/status >/dev/null +sleep 2 +remote_code="$( + ssh -o BatchMode=yes -o ConnectTimeout=10 "${MEMIND_PORTAL_TUNNEL_HOST}" \ + "curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}/api/status" \ + 2>/dev/null || true +)" +[[ "${remote_code}" == "200" ]] +printf 'stable_port=8081\n' +printf 'remote_tunnel_status=%s\n' "${remote_code}" +REMOTE + +echo "103 canary rollback completed; stable Portal remains active on 8081." diff --git a/scripts/run-memind-canary-proxy-prod.sh b/scripts/run-memind-canary-proxy-prod.sh new file mode 100755 index 0000000..be07119 --- /dev/null +++ b/scripts/run-memind-canary-proxy-prod.sh @@ -0,0 +1,35 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +STABLE_ROOT="${MEMIND_CANARY_STABLE_ROOT:-/Users/john/Project/Memind}" +SECRET_FILE="${MEMIND_CANARY_SECRET_FILE:-${HOME}/.config/memind/canary-router.secret}" + +if [[ ! -d "${STABLE_ROOT}" || ! -f "${STABLE_ROOT}/.env" ]]; then + echo "[canary-proxy] stable persistent root is unavailable: ${STABLE_ROOT}" >&2 + exit 1 +fi +if [[ ! -r "${SECRET_FILE}" ]]; then + echo "[canary-proxy] diagnostic secret file is unavailable: ${SECRET_FILE}" >&2 + exit 1 +fi + +set -a +# shellcheck disable=SC1091 +source "${STABLE_ROOT}/.env" +set +a + +export NODE_ENV=production +export MEMIND_PORTAL_H5_ROOT="${STABLE_ROOT}" +export MEMIND_CANARY_DIAGNOSTIC_SECRET="$(tr -d '\r\n' < "${SECRET_FILE}")" +export MEMIND_CANARY_PROXY_HOST=127.0.0.1 +export MEMIND_CANARY_PROXY_PORT="${MEMIND_CANARY_PROXY_PORT:-18080}" +export MEMIND_CANARY_STABLE_URL="${MEMIND_CANARY_STABLE_URL:-http://127.0.0.1:8081}" +export MEMIND_CANARY_CANDIDATE_URL="${MEMIND_CANARY_CANDIDATE_URL:-http://127.0.0.1:18081}" + +NODE_BIN="${NODE_BIN:-/opt/homebrew/opt/node@24/bin/node}" +if [[ ! -x "${NODE_BIN}" ]]; then + NODE_BIN="$(command -v node)" +fi + +exec "${NODE_BIN}" "${ROOT}/memind-canary-proxy.mjs" diff --git a/scripts/run-memind-portal-candidate.sh b/scripts/run-memind-portal-candidate.sh new file mode 100755 index 0000000..dc5f21b --- /dev/null +++ b/scripts/run-memind-portal-candidate.sh @@ -0,0 +1,43 @@ +#!/usr/bin/env bash +set -euo pipefail + +ROOT="$(cd "$(dirname "$0")/.." && pwd)" +STABLE_ROOT="${MEMIND_CANARY_STABLE_ROOT:-/Users/john/Project/Memind}" + +if [[ ! -d "${STABLE_ROOT}" || ! -f "${STABLE_ROOT}/.env" ]]; then + echo "[portal-candidate] stable persistent root is unavailable: ${STABLE_ROOT}" >&2 + exit 1 +fi + +set -a +# shellcheck disable=SC1091 +source "${STABLE_ROOT}/.env" +set +a + +export NODE_ENV=production +export MEMIND_PORTAL_H5_ROOT="${STABLE_ROOT}" +export MEMIND_RUNTIME_ROLE=candidate +export MEMIND_CANARY_PASSIVE_RUNTIME=1 +export MEMIND_RUNTIME_BUILD_ID="${MEMIND_CANARY_RELEASE_ID:?MEMIND_CANARY_RELEASE_ID is required}" +export MEMIND_AGENT_RUN_WORKER_ID="portal-candidate-${MEMIND_RUNTIME_BUILD_ID}" +export MEMIND_WORKSPACE_MAINTENANCE=0 +export H5_REMINDER_WORKER_ENABLED=0 +export H5_PORT="${MEMIND_CANARY_CANDIDATE_PORT:-18081}" +export H5_HOST=127.0.0.1 +export H5_PUBLIC_BASE_URL="${H5_PUBLIC_BASE_URL:-https://m.tkmind.cn}" +export TKMIND_API_TARGETS="${MEMIND_CANARY_GOOSED_URL:-https://127.0.0.1:18015}" +export TKMIND_API_TARGET="${MEMIND_CANARY_GOOSED_URL:-https://127.0.0.1:18015}" +unset TKMIND_API_TARGET_1 + +NODE_BIN="${NODE_BIN:-/opt/homebrew/opt/node@24/bin/node}" +if [[ ! -x "${NODE_BIN}" ]]; then + NODE_BIN="$(command -v node)" +fi + +port="${H5_PORT}" +if lsof -tiTCP:"${port}" -sTCP:LISTEN >/dev/null 2>&1; then + echo "[portal-candidate] port ${port} is already in use; refusing to replace it" >&2 + exit 1 +fi + +exec "${NODE_BIN}" "${ROOT}/server.mjs" diff --git a/server/portal-agent-services-bootstrap.mjs b/server/portal-agent-services-bootstrap.mjs index 601a7e3..5f0bf99 100644 --- a/server/portal-agent-services-bootstrap.mjs +++ b/server/portal-agent-services-bootstrap.mjs @@ -8,6 +8,7 @@ import { createMindSpaceAuditWriter } from '../mindspace-audit.mjs'; import { createMindSpaceImageGenerationService } from '../mindspace-image-generation.mjs'; import { createMindSpaceImageReviewService } from '../mindspace-image-review.mjs'; import { createWordFilterService } from '../word-filter.mjs'; +import { isPassiveCanaryRuntime } from './portal-runtime-role.mjs'; async function createPgExperienceServiceFromModule(options) { const { createPgExperienceService } = await import( @@ -108,28 +109,32 @@ export async function bootstrapPortalAgentServices({ const mindSpaceAudit = createMindSpaceAuditWriterFn(pool); - mindSpaceRuntimeAdapter.startBackgroundJobs({ - publicationCleanupIntervalMs: 60 * 1000, - agentWorker: runtime.agentWorker, - agentRunner: mindSpaceAgentRunner, - workspaceMaintenanceEnabled, - publishRoot: runtime.publishRoot, - startWorkspaceThumbnailWatcher, - startWorkspaceAssetSyncWatcher, - syncUserWorkspaceByDirKey: async ( - dirKey, - options, - ) => { - const userId = - await resolveUserIdByDirKey(dirKey); - if (!userId) return; - await mindSpaceAssets.syncWorkspaceAssets( - userId, + if (isPassiveCanaryRuntime(env)) { + logger.log('Passive candidate runtime: MindSpace background jobs disabled'); + } else { + mindSpaceRuntimeAdapter.startBackgroundJobs({ + publicationCleanupIntervalMs: 60 * 1000, + agentWorker: runtime.agentWorker, + agentRunner: mindSpaceAgentRunner, + workspaceMaintenanceEnabled, + publishRoot: runtime.publishRoot, + startWorkspaceThumbnailWatcher, + startWorkspaceAssetSyncWatcher, + syncUserWorkspaceByDirKey: async ( + dirKey, options, - ); - }, - expireStaleUploadsIntervalMs: 5 * 60 * 1000, - }); + ) => { + const userId = + await resolveUserIdByDirKey(dirKey); + if (!userId) return; + await mindSpaceAssets.syncWorkspaceAssets( + userId, + options, + ); + }, + expireStaleUploadsIntervalMs: 5 * 60 * 1000, + }); + } await userAuth.ensureAdminUser(); const userDataSpaceBackfill = diff --git a/server/portal-agent-services-bootstrap.test.mjs b/server/portal-agent-services-bootstrap.test.mjs index 2bd5e5a..4e4cd8f 100644 --- a/server/portal-agent-services-bootstrap.test.mjs +++ b/server/portal-agent-services-bootstrap.test.mjs @@ -424,3 +424,25 @@ test('can disable relay bootstrap for backend-LLM-only isolated runs', async () assert.equal(setup.calls.some(([name]) => name === 'relay-bootstrap'), false); assert.equal(setup.calls.some(([name]) => name === 'provider-sync'), true); }); + +test('passive candidate runtime does not start MindSpace background jobs', async () => { + const setup = createSetup({ + env: { + EXPERIENCE_PG_URL: 'postgres://experience', + MEMIND_RUNTIME_ROLE: 'candidate', + MEMIND_CANARY_PASSIVE_RUNTIME: '1', + }, + }); + + const result = await bootstrapPortalAgentServices(setup.options); + await Promise.all([result.relayBootstrapTask, result.providerSyncTask]); + + assert.equal(setup.calls.some(([name]) => name === 'start-background'), false); + assert.ok( + setup.calls.some( + ([name, message]) => + name === 'log' + && message === 'Passive candidate runtime: MindSpace background jobs disabled', + ), + ); +}); diff --git a/server/portal-domain-services-bootstrap.mjs b/server/portal-domain-services-bootstrap.mjs index 1d68b58..d438257 100644 --- a/server/portal-domain-services-bootstrap.mjs +++ b/server/portal-domain-services-bootstrap.mjs @@ -23,6 +23,7 @@ import { createPlazaOpsService } from '../plaza-ops.mjs'; import { createPlazaPostService, formatPostRow } from '../plaza-posts.mjs'; import { createPlazaRecommendService } from '../plaza-recommend.mjs'; import { createPlazaRedis } from '../plaza-redis.mjs'; +import { isPassiveCanaryRuntime } from './portal-runtime-role.mjs'; import { createPlazaSeoService } from '../plaza-seo.mjs'; import { startPlazaTasks, @@ -287,12 +288,16 @@ export async function bootstrapPortalDomainServices({ invalidateFeedCaches: () => plazaRedis?.invalidateFeedCaches?.(), }); - startPlazaTasksFn({ - pool, - plazaRedis, - recalculateHotScores: recalculateHotScoresFn, - writebackPublications: writebackPublicationsFn, - }); + if (isPassiveCanaryRuntime(env)) { + logger.log('Passive candidate runtime: Plaza background tasks disabled'); + } else { + startPlazaTasksFn({ + pool, + plazaRedis, + recalculateHotScores: recalculateHotScoresFn, + writebackPublications: writebackPublicationsFn, + }); + } return { mindSearchConfigService, diff --git a/server/portal-domain-services-bootstrap.test.mjs b/server/portal-domain-services-bootstrap.test.mjs index 72a7f09..35c872c 100644 --- a/server/portal-domain-services-bootstrap.test.mjs +++ b/server/portal-domain-services-bootstrap.test.mjs @@ -416,3 +416,24 @@ test('domain bootstrap preserves directory and Plaza cross-service callbacks', a ], ); }); + +test('passive candidate runtime does not start Plaza background tasks', async () => { + const setup = createBootstrapSetup({ + env: { + H5_DEFAULT_TIMEZONE: 'Asia/Singapore', + MEMIND_RUNTIME_ROLE: 'candidate', + MEMIND_CANARY_PASSIVE_RUNTIME: '1', + }, + }); + + await bootstrapPortalDomainServices(setup.dependencies); + + assert.equal(setup.calls.some((call) => call.kind === 'plaza-tasks'), false); + assert.ok( + setup.calls.some( + (call) => + call.kind === 'log' + && call.message === 'Passive candidate runtime: Plaza background tasks disabled', + ), + ); +}); diff --git a/server/portal-gateway-services-bootstrap.mjs b/server/portal-gateway-services-bootstrap.mjs index 4a68abe..a80c9c8 100644 --- a/server/portal-gateway-services-bootstrap.mjs +++ b/server/portal-gateway-services-bootstrap.mjs @@ -18,6 +18,7 @@ import { createOrchestratorAdminConfigService } from '../services/orchestrator/a import { createWorkflowShadowObserver } from '../services/orchestrator/shadow-observer.mjs'; import { createTkmindProxy } from '../tkmind-proxy.mjs'; import { createToolGateway } from '../tool-gateway.mjs'; +import { isPassiveCanaryRuntime } from './portal-runtime-role.mjs'; function isEnabledFlag(value, fallback = '') { return ['1', 'true', 'yes', 'on'].includes( @@ -433,11 +434,12 @@ export function bootstrapPortalGatewayServices({ ), workerIdentity, }); - const agentRunRecoveryTimer = - startAgentRunRecoveryLoopFn( - agentRunGateway, - { env, logger: console }, - ); + const agentRunRecoveryTimer = isPassiveCanaryRuntime(env) + ? null + : startAgentRunRecoveryLoopFn( + agentRunGateway, + { env, logger: console }, + ); return { tkmindProxy, diff --git a/server/portal-gateway-services-bootstrap.test.mjs b/server/portal-gateway-services-bootstrap.test.mjs index 11602ff..157e6e1 100644 --- a/server/portal-gateway-services-bootstrap.test.mjs +++ b/server/portal-gateway-services-bootstrap.test.mjs @@ -288,6 +288,25 @@ test('preserves memory, page sync, and busy callbacks', async () => { ); }); +test('passive candidate runtime disables the singleton Agent recovery loop', () => { + let recoveryStarts = 0; + const setup = createSetup({ + env: { + MEMIND_RUNTIME_ROLE: 'candidate', + MEMIND_CANARY_PASSIVE_RUNTIME: '1', + }, + startAgentRunRecoveryLoopFn() { + recoveryStarts += 1; + return { id: 'recovery' }; + }, + }); + + const result = bootstrapPortalGatewayServices(setup.options); + + assert.equal(recoveryStarts, 0); + assert.equal(result.agentRunRecoveryTimer, null); +}); + test('keeps memory observation optional and parses disabled dispatch', async () => { const setup = createSetup({ env: { diff --git a/server/portal-integration-services-bootstrap.mjs b/server/portal-integration-services-bootstrap.mjs index fc76e4c..0d97001 100644 --- a/server/portal-integration-services-bootstrap.mjs +++ b/server/portal-integration-services-bootstrap.mjs @@ -11,6 +11,7 @@ import { createPageEditSessionService } from '../mindspace-page-edit-session.mjs import { resolveMindSpaceRuntimeConfig } from '../mindspace-runtime-config.mjs'; import { createNotificationDispatcher } from '../notification-dispatcher.mjs'; import { startScheduleReminderWorker } from '../schedule-reminder-worker.mjs'; +import { isPassiveCanaryRuntime } from './portal-runtime-role.mjs'; import { loadWechatMpModule } from '../wechat-mp-loader.mjs'; export async function bootstrapPortalIntegrationServices({ @@ -224,6 +225,7 @@ export async function bootstrapPortalIntegrationServices({ let scheduleReminderWorker = null; if ( + !isPassiveCanaryRuntime(env) && env.H5_REMINDER_WORKER_ENABLED === '1' && wechatMpService?.enabled && scheduleService @@ -237,7 +239,7 @@ export async function bootstrapPortalIntegrationServices({ } let subscriptionExpiryTimer = null; - if (subscriptionService) { + if (subscriptionService && !isPassiveCanaryRuntime(env)) { subscriptionExpiryTimer = setIntervalFn( async () => { try { diff --git a/server/portal-integration-services-bootstrap.test.mjs b/server/portal-integration-services-bootstrap.test.mjs index 381ca83..cbd28b9 100644 --- a/server/portal-integration-services-bootstrap.test.mjs +++ b/server/portal-integration-services-bootstrap.test.mjs @@ -507,3 +507,24 @@ test('keeps optional integrations disabled and contains timer failures', async ( ), ); }); + +test('passive candidate runtime disables singleton reminder and subscription timers', async () => { + const setup = createSetup({ + env: { + H5_SCHEDULE_ENABLED: '1', + H5_REMINDER_WORKER_ENABLED: '1', + MEMIND_RUNTIME_ROLE: 'candidate', + MEMIND_CANARY_PASSIVE_RUNTIME: '1', + }, + }); + + const result = await bootstrapPortalIntegrationServices(setup.options); + const captured = setup.getCaptured(); + + assert.equal(result.scheduleReminderWorker, null); + assert.equal(result.subscriptionExpiryTimer, null); + assert.equal(captured.reminderOptions, undefined); + assert.equal(captured.timerCallback, undefined); + assert.equal(setup.calls.some(([name]) => name === 'reminder-worker'), false); + assert.equal(setup.calls.some(([name]) => name === 'set-interval'), false); +}); diff --git a/server/portal-runtime-role.mjs b/server/portal-runtime-role.mjs new file mode 100644 index 0000000..3a785a5 --- /dev/null +++ b/server/portal-runtime-role.mjs @@ -0,0 +1,13 @@ +const RUNTIME_ROLES = new Set(['stable', 'candidate']); + +export function resolvePortalRuntimeRole(env = process.env) { + const configured = String(env.MEMIND_RUNTIME_ROLE ?? '').trim().toLowerCase(); + return RUNTIME_ROLES.has(configured) ? configured : 'stable'; +} + +export function isPassiveCanaryRuntime(env = process.env) { + return ( + resolvePortalRuntimeRole(env) === 'candidate' + && String(env.MEMIND_CANARY_PASSIVE_RUNTIME ?? '1').trim() !== '0' + ); +} diff --git a/server/portal-runtime-role.test.mjs b/server/portal-runtime-role.test.mjs new file mode 100644 index 0000000..ed8c875 --- /dev/null +++ b/server/portal-runtime-role.test.mjs @@ -0,0 +1,30 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; + +import { + isPassiveCanaryRuntime, + resolvePortalRuntimeRole, +} from './portal-runtime-role.mjs'; + +test('runtime role defaults to stable and rejects unknown values', () => { + assert.equal(resolvePortalRuntimeRole({}), 'stable'); + assert.equal(resolvePortalRuntimeRole({ MEMIND_RUNTIME_ROLE: 'unknown' }), 'stable'); +}); + +test('candidate runtime is passive unless explicitly disabled', () => { + assert.equal( + isPassiveCanaryRuntime({ MEMIND_RUNTIME_ROLE: 'candidate' }), + true, + ); + assert.equal( + isPassiveCanaryRuntime({ + MEMIND_RUNTIME_ROLE: 'candidate', + MEMIND_CANARY_PASSIVE_RUNTIME: '0', + }), + false, + ); + assert.equal( + isPassiveCanaryRuntime({ MEMIND_RUNTIME_ROLE: 'stable' }), + false, + ); +}); diff --git a/server/portal-runtime-routes.mjs b/server/portal-runtime-routes.mjs index 0ea862c..3806f9c 100644 --- a/server/portal-runtime-routes.mjs +++ b/server/portal-runtime-routes.mjs @@ -1,3 +1,5 @@ +import { resolvePortalRuntimeRole } from './portal-runtime-role.mjs'; + function runtimeEnvFlag(value, fallback = false) { const raw = String(value ?? '').trim().toLowerCase(); if (!raw) return fallback; @@ -54,6 +56,7 @@ export function attachPortalRuntimeRoutes( } api.get('/status', async (_req, res, next) => { + res.setHeader?.('X-Memind-Runtime-Role', resolvePortalRuntimeRole(env)); await waitForUserAuthReady(); const userAuth = getUserAuth(); const tkmindProxy = getTkmindProxy(); diff --git a/server/portal-runtime-routes.test.mjs b/server/portal-runtime-routes.test.mjs index 099b536..ae0fc4a 100644 --- a/server/portal-runtime-routes.test.mjs +++ b/server/portal-runtime-routes.test.mjs @@ -14,6 +14,7 @@ function createRouterRecorder() { function createResponseRecorder() { return { + headers: new Map(), statusCode: 200, body: undefined, responseType: null, @@ -31,6 +32,10 @@ function createResponseRecorder() { this.body = body; return this; }, + setHeader(name, value) { + this.headers.set(String(name).toLowerCase(), String(value)); + return this; + }, }; } @@ -73,6 +78,31 @@ test('GET /status preserves upstream status and body', async () => { assert.equal(res.statusCode, 207); assert.equal(res.responseType, 'send'); assert.equal(res.body, 'upstream-status'); + assert.equal(res.headers.get('x-memind-runtime-role'), 'stable'); +}); + +test('GET /status identifies a passive candidate runtime without changing the body', async () => { + const api = createRouterRecorder(); + attachPortalRuntimeRoutes(api, { + env: { MEMIND_RUNTIME_ROLE: 'candidate' }, + getUserAuth: () => ({}), + getTkmindProxy: () => ({ + async apiFetch() { + return { + status: 200, + async text() { + return 'ok'; + }, + }; + }, + }), + }); + const res = createResponseRecorder(); + + await api.routes.get('/status')({}, res, () => {}); + + assert.equal(res.body, 'ok'); + assert.equal(res.headers.get('x-memind-runtime-role'), 'candidate'); }); test('GET /status falls through when multi-user proxy services are unavailable', async () => {