feat: add guarded portal canary release
Memind CI / Test, build, and release guards (push) Failing after 2m14s
Memind CI / Test, build, and release guards (push) Failing after 2m14s
This commit is contained in:
john
@@ -57,6 +57,9 @@ jobs:
|
||||
- name: Run full test suite
|
||||
run: npm test
|
||||
|
||||
- name: Verify release gate and canary routing
|
||||
run: npm run test:release-gate:unit
|
||||
|
||||
- name: Build production frontend
|
||||
run: npm run build
|
||||
|
||||
|
||||
@@ -55,7 +55,7 @@ git rebase origin/main
|
||||
1. 本机不允许直接 `rsync` 到 `103` 或 `105`。
|
||||
2. **禁止** SSH 登录 `105` 后直接修改业务源码（含 `scripts/wechat-mp-menu.mjs` 等）；必须先本地 commit，再按发布流程上线。详见 [105 服务器变更规范](docs/105-server-operations.md)。
|
||||
3. Portal 生产与测试统一走“本机构建 runtime artifact -> 打包发布”，**禁止**在 `103` 解源码包后 `npm install` / `npm run build`。
|
||||
4. Portal 唯一合法生产入口是 `bash scripts/release-portal-runtime-prod.sh`。
|
||||
4. Portal 首次生产入口是 `bash scripts/release-portal-canary-prod.sh`；它只安装并启用用户级灰度候选。`bash scripts/release-portal-runtime-prod.sh` 是整包晋升脚本，在同一候选完成灰度验收且晋升证据校验落地前继续禁止非 dry-run。
|
||||
5. 发布只能从已验证的完整 `main` 打整包；分支代码必须先合并进 `main`，禁止从功能分支、单个 commit、单个修复或局部差异单包发布。
|
||||
6. 发布前必须确认 CI 已通过，且没有未合并的关键变更。
|
||||
7. 发布来源必须是可追溯 commit，不允许从不明工作区直接出包。
|
||||
|
||||
@@ -8,7 +8,7 @@
|
||||
2. **禁止 SSH 登录 `105` 直接修改业务代码**（含服务号菜单脚本 `scripts/wechat-mp-menu.mjs`）。105 上文件是部署产物；变更必须：本地 `test-memind` 修改 → Git commit → 正式发布 → 必要时在目标环境执行 API 同步。详见 [docs/105-server-operations.md](docs/105-server-operations.md)。
|
||||
2. **Portal 生产必须是无源码 runtime 模式**：构建只发生在本机 Mac，产物是 `.runtime/portal/`；`103` 只接收 runtime artifact、继承持久目录、启动服务，**禁止**在 `103` 上 `npm install`、`npm run build` 或保留可运行源码树。
|
||||
2. `MindSpace` 独立服务同样必须走单独 runtime artifact：本地 `node scripts/build-mindspace-service-runtime.mjs` -> `bash scripts/release-mindspace-service-prod.sh` -> 上传 `103` -> 备份 `/Users/john/MindSpace` 与共享 `Memind/.env` -> 原子切换到 `/Users/john/MindSpace` -> 健康检查 `127.0.0.1:8082/health` 与 `/mindspace/v1/contract`；禁止手工 SSH 改线上 `/Users/john/MindSpace` 源码。103 当前拓扑见 [docs/103-runtime-topology.md](docs/103-runtime-topology.md)。
|
||||
3. Portal 生产发布必须先构建候选 runtime 并通过用户级灰度入口；当前 `scripts/release-portal-runtime-prod.sh` 是整包替换脚本，在用户级灰度入口未上线、未验证前禁止调用。灰度只能命中明确的不可变用户身份，未命中请求必须继续走稳定版本，禁止直接全量切换。
|
||||
3. Portal 生产发布必须先构建候选 runtime，并只通过 `scripts/release-portal-canary-prod.sh` 启用用户级灰度。灰度只能命中明确的不可变用户身份，未命中、身份解析失败或候选不健康必须继续走稳定版本。`scripts/release-portal-runtime-prod.sh` 仍是整包替换脚本，在同一候选的灰度验收和晋升证据校验完成前禁止非 dry-run。
|
||||
4. `scripts/release-prod.sh`（源码包发布）已停用，不得再用于 Portal；`rsync_to_server.sh` 与任何面向 `105` 的直接同步脚本也只保留为禁用提示。
|
||||
5. Portal 发布包不得携带运行态资产；`.env`、`data/`、`users/`、`.tailscale/`、`public/plaza-covers/`、`logs/` 只能从线上现有 live 目录继承。`/Users/john/MindSpace` 是独立 MindSpace Service 的生产根目录，不属于 Portal runtime 包；`/Users/john/Project/Memind/MindSpace` 只允许作为旧链路兼容/存量目录处理，不得再被写成 MindSpace Service 的当前根目录。
|
||||
6. runtime artifact 必须包含 `server.mjs` 与 `mindspace-sandbox-mcp.mjs`（`sandbox-fs` 扩展依赖的独立子进程入口）。
|
||||
|
||||
@@ -35,6 +35,19 @@ Local Colima may exist for other workloads, but local Memind/goosed is not suppo
|
||||
|
||||
Portal must be released as a runtime artifact. Do not edit source directly on 103.
|
||||
|
||||
### User-level canary (when enabled)
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Identity router | `127.0.0.1:18080` |
|
||||
| Candidate Portal | `127.0.0.1:18081` |
|
||||
| Candidate goosed | `127.0.0.1:18015` |
|
||||
| Release entry | `scripts/release-portal-canary-prod.sh` |
|
||||
| Rollback entry | `scripts/rollback-portal-canary-prod.sh` |
|
||||
|
||||
The stable Portal stays on 8081. The 105 reverse tunnel points to 18080 only after the
|
||||
candidate, router, backups, immutable identity checks, and fallback checks pass.
|
||||
|
||||
## MindSpace Service
|
||||
|
||||
| Item | Value |
|
||||
|
||||
@@ -544,9 +544,18 @@ failed = skipped = blocked = unknown = cleanup_failed = 0
|
||||
|
||||
### 10.0 用户级灰度前置条件
|
||||
|
||||
当前 Portal 发布脚本会替换 8081 live 目录，不具备版本级用户路由能力，因此在用户级灰度代理和候选 runtime 并行运行方案完成前，禁止调用该脚本执行生产发布。灰度身份必须使用不可变用户 ID/用户名/微信绑定 ID；禁止按显示昵称（包括“唐”）匹配。未命中或身份解析失败必须回落稳定版本，候选不可用必须自动回退。
|
||||
用户级灰度入口由 `scripts/release-portal-canary-prod.sh` 安装：稳定 Portal 保持 8081，
|
||||
候选 Portal 使用 18081，候选 goosed 使用 18015，身份路由器使用 18080。灰度身份必须
|
||||
使用不可变用户 ID、唯一用户名或微信绑定账号标识；禁止按显示昵称（包括“唐”）匹配。未命中、
|
||||
身份解析失败或候选不可用必须自动回落稳定版本。
|
||||
|
||||
本地路由契约见 [docs/release-canary-103.md](release-canary-103.md)，对应测试为 `release-gate/canary-routing.test.mjs`。该测试通过只代表身份匹配契约通过，不代表 103 已具备灰度入口。
|
||||
完整契约和回滚入口见 [103 用户级灰度发布](release-canary-103.md)，对应测试至少包括
|
||||
`release-gate/canary-routing.test.mjs`、`memind-canary-proxy.test.mjs` 和
|
||||
`release-gate/release-script.test.mjs`。测试通过不代表已经部署；103 灰度仍须绑定同一
|
||||
commit、artifact 和 Gate report，并完成备份、只读预检及人工批准。
|
||||
|
||||
`scripts/release-portal-runtime-prod.sh` 会替换 8081 live 目录。它只用于同一候选完成灰度
|
||||
验收后的全量晋升；在晋升证据校验完成前继续硬阻断非 dry-run，不能拿灰度入口替代晋升批准。
|
||||
|
||||
`scripts/release-portal-runtime-prod.sh` 必须按以下顺序执行：
|
||||
|
||||
|
||||
@@ -1,13 +1,71 @@
|
||||
# 103 用户级灰度发布设计
|
||||
# 103 用户级灰度发布
|
||||
|
||||
当前 `scripts/release-portal-runtime-prod.sh` 是整包替换脚本，不能用于“只让 john 和精确微信用户唐”灰度。因此在闸门通过前禁止调用它进行生产发布。
|
||||
Portal 的首个生产动作必须是用户级灰度，不能直接整包替换稳定版本。
|
||||
|
||||
候选版本必须与稳定版本并行运行，由入口代理依据登录后的不可变身份选择版本：
|
||||
## 运行拓扑
|
||||
|
||||
- `john`：按唯一用户名或数据库用户 ID 匹配。
|
||||
- 微信用户“唐”：按绑定的不可变用户 ID `wx_ul610et8` 匹配；禁止按昵称匹配，避免与其他同名用户串流。
|
||||
- 未命中、身份缺失、解析异常：全部回落到稳定版本。
|
||||
| 组件 | 地址 | 说明 |
|
||||
|---|---|---|
|
||||
| 稳定 Portal | `127.0.0.1:8081` | 保持当前 live 目录和进程，不在灰度发布中替换 |
|
||||
| 灰度路由器 | `127.0.0.1:18080` | 105 的 `19081` 反向隧道在全部检查通过后切到这里 |
|
||||
| 候选 Portal | `127.0.0.1:18081` | 独立、无源码 runtime；使用被动运行时模式 |
|
||||
| 候选 goosed | `127.0.0.1:18015` | 独立容器并挂载候选 runtime，不重建稳定 9 个容器 |
|
||||
|
||||
候选版本必须使用独立监听端口和独立 runtime 目录，但共享经审计的持久数据与数据库连接；灰度入口必须支持 SSE/附件/公开页请求的完整转发，并保留稳定版本回退。
|
||||
候选 Portal 通过 `MEMIND_PORTAL_H5_ROOT` 共享经审计的持久数据和数据库，但
|
||||
`MEMIND_RUNTIME_ROLE=candidate` 会禁止重复启动 MindSpace、Plaza、提醒、订阅续期和
|
||||
Agent 恢复扫描等单例后台任务。
|
||||
|
||||
上线前必须在本地证明：命中用户只到候选版本、其他用户只到稳定版本、同名非目标用户不命中、身份解析失败回落稳定、候选不可用自动回退。生产发布仍需在 `main` 干净提交、187 场景全通过并获得单独的生产灰度确认后执行；本设计不会触发生产动作。
|
||||
## 身份和回落规则
|
||||
|
||||
- `john`：按数据库中的唯一用户名或用户 ID 匹配。
|
||||
- 微信用户“唐”：只按绑定微信身份的唯一账号标识 `wx_ul610et8` 匹配；该值在生产库中是唯一 `username/slug`，不是显示昵称。
|
||||
- 显示昵称永远不参与路由。
|
||||
- Cookie 会话通过 `h5_login_sessions` 的 token hash 只读解析。
|
||||
- 微信回调通过 `FromUserName` 和 `h5_user_wechat_identities` 只读解析到绑定账号，再用唯一账号标识匹配。
|
||||
- 未命中、无身份、过期/撤销会话、数据库异常：走稳定版本。
|
||||
- 候选 `/api/status` 不健康：自动走稳定版本。
|
||||
- SSE、附件、公开页和普通 API 均由同一反向代理转发；微信 XML body 必须原样保留。
|
||||
|
||||
## 发布入口
|
||||
|
||||
候选必须来自干净且等于 `origin/main` 的完整 `main`，CI 成功，并具有与
|
||||
`.runtime/portal` 完全一致、未过期的 187 项 Gate report。
|
||||
|
||||
```bash
|
||||
bash scripts/release-portal-canary-prod.sh --yes
|
||||
```
|
||||
|
||||
该脚本按顺序执行：
|
||||
|
||||
1. 验证源码、分支、远端主线、Gate report 和 artifact hash。
|
||||
2. 执行 103 只读预检。
|
||||
3. 上传经过验证的候选包。
|
||||
4. 创建稳定 live 全量备份和持久目录备份，并校验 SHA256、gzip 和关键目录。
|
||||
5. 启动独立候选 goosed、候选 Portal 和灰度路由器。
|
||||
6. 验证不可变身份配置、稳定/候选健康和路由器诊断。
|
||||
7. 短暂阻止新 Agent Run，排空活动任务。
|
||||
8. 只把反向隧道从 8081 切到 18080。
|
||||
9. 主动停止候选，证明目标身份会自动回落稳定版，再恢复候选。
|
||||
|
||||
任一步失败都会把隧道恢复到稳定 8081，停止候选服务并保留候选目录和备份证据。
|
||||
|
||||
## 回滚
|
||||
|
||||
```bash
|
||||
bash scripts/rollback-portal-canary-prod.sh --yes
|
||||
```
|
||||
|
||||
回滚只执行以下动作：
|
||||
|
||||
- 把 105 反向隧道恢复到稳定 Portal 8081。
|
||||
- 停止灰度路由器、候选 Portal 和候选 goosed。
|
||||
- 清除发布排水标记。
|
||||
- 验证稳定 Portal 和 105 隧道健康。
|
||||
|
||||
不会删除候选目录、备份或稳定运行目录。
|
||||
|
||||
## 全量晋升
|
||||
|
||||
`scripts/release-portal-runtime-prod.sh` 仍是整包替换脚本，不能把“灰度路由器已实现”
|
||||
解释为自动获得全量切换权限。全量晋升必须使用同一 commit 和 artifact，保留灰度验收
|
||||
证据，并再次取得明确批准；在晋升入口完成对应证据校验前，该脚本继续硬阻断非 dry-run。
|
||||
|
||||
+5
-3
@@ -45,13 +45,15 @@ npm run check:mindspace-public-links
|
||||
|
||||
## 2. Portal runtime 发布唯一流程
|
||||
|
||||
生产发布守门员落地、完整 Gate report 通过且获得本次明确人工批准后，只使用：
|
||||
生产发布守门员、完整 Gate report 和本次明确人工批准全部满足后，首次生产动作只使用：
|
||||
|
||||
```bash
|
||||
bash scripts/release-portal-runtime-prod.sh --yes
|
||||
bash scripts/release-portal-canary-prod.sh --yes
|
||||
```
|
||||
|
||||
生产 `103` 禁止使用 `--skip-tests` 或任何范围绕过变量。本文历史记录中的旧绕过命令只用于事故复盘，不构成当前发布授权。
|
||||
生产 `103` 禁止使用 `--skip-tests` 或任何范围绕过变量。整包替换脚本
|
||||
`release-portal-runtime-prod.sh` 只用于同一候选完成灰度验收后的全量晋升，在晋升证据
|
||||
校验完成前仍会硬阻断。本文历史记录中的旧绕过命令只用于事故复盘，不构成当前发布授权。
|
||||
|
||||
这个脚本必须完成这些动作：
|
||||
|
||||
|
||||
@@ -0,0 +1,569 @@
|
||||
#!/usr/bin/env node
|
||||
import crypto from 'node:crypto';
|
||||
import http from 'node:http';
|
||||
import { pathToFileURL } from 'node:url';
|
||||
|
||||
import mysql from 'mysql2/promise';
|
||||
|
||||
import {
|
||||
canaryPolicyFromEnv,
|
||||
resolveCanaryTarget,
|
||||
} from './release-gate/canary-routing.mjs';
|
||||
import { loadMemindEnvFiles } from './scripts/memind-runtime-profile.mjs';
|
||||
|
||||
const SESSION_COOKIE = 'tkmind_user_session';
|
||||
const DEFAULT_BODY_LIMIT = 2 * 1024 * 1024;
|
||||
const RETRYABLE_CONNECT_ERRORS = new Set([
|
||||
'ECONNREFUSED',
|
||||
'EHOSTUNREACH',
|
||||
'ENETUNREACH',
|
||||
]);
|
||||
|
||||
function csvValues(value) {
|
||||
return String(value ?? '')
|
||||
.split(',')
|
||||
.map((item) => item.trim())
|
||||
.filter(Boolean);
|
||||
}
|
||||
|
||||
export function parseCookieHeader(header) {
|
||||
const cookies = new Map();
|
||||
for (const item of String(header ?? '').split(';')) {
|
||||
const separator = item.indexOf('=');
|
||||
if (separator < 1) continue;
|
||||
const name = item.slice(0, separator).trim();
|
||||
const rawValue = item.slice(separator + 1).trim();
|
||||
if (!name) continue;
|
||||
try {
|
||||
cookies.set(name, decodeURIComponent(rawValue));
|
||||
} catch {
|
||||
cookies.set(name, rawValue);
|
||||
}
|
||||
}
|
||||
return cookies;
|
||||
}
|
||||
|
||||
export function parseWechatOpenid(xml) {
|
||||
const match = String(xml ?? '').match(
|
||||
/<FromUserName><!\[CDATA\[([\s\S]*?)\]\]><\/FromUserName>/i,
|
||||
);
|
||||
return String(match?.[1] ?? '').trim();
|
||||
}
|
||||
|
||||
function sessionTokenHash(token) {
|
||||
return crypto.createHash('sha256').update(token).digest('hex');
|
||||
}
|
||||
|
||||
function normalizeLoopbackTarget(value, label) {
|
||||
const target = new URL(String(value ?? ''));
|
||||
if (
|
||||
target.protocol !== 'http:'
|
||||
|| !['127.0.0.1', 'localhost', '::1'].includes(target.hostname)
|
||||
) {
|
||||
throw new Error(`${label} must be a loopback http URL`);
|
||||
}
|
||||
target.pathname = target.pathname.replace(/\/+$/, '');
|
||||
target.search = '';
|
||||
target.hash = '';
|
||||
return target;
|
||||
}
|
||||
|
||||
function createIdentityPool(env = process.env) {
|
||||
const poolOptions = {
|
||||
waitForConnections: true,
|
||||
connectionLimit: Math.max(
|
||||
1,
|
||||
Math.min(8, Number(env.MEMIND_CANARY_DB_POOL_SIZE ?? 4)),
|
||||
),
|
||||
queueLimit: 32,
|
||||
};
|
||||
if (env.DATABASE_URL) {
|
||||
return mysql.createPool({ uri: env.DATABASE_URL, ...poolOptions });
|
||||
}
|
||||
if (!env.MYSQL_HOST || !env.MYSQL_DATABASE) {
|
||||
throw new Error('Canary identity routing requires DATABASE_URL or MYSQL_*');
|
||||
}
|
||||
return mysql.createPool({
|
||||
host: env.MYSQL_HOST,
|
||||
port: Number(env.MYSQL_PORT ?? 3306),
|
||||
user: env.MYSQL_USER ?? 'boot',
|
||||
password: env.MYSQL_PASSWORD ?? '',
|
||||
database: env.MYSQL_DATABASE,
|
||||
...poolOptions,
|
||||
});
|
||||
}
|
||||
|
||||
export function createCanaryIdentityResolver({
|
||||
pool,
|
||||
appId = '',
|
||||
now = () => Date.now(),
|
||||
} = {}) {
|
||||
if (!pool?.query) throw new Error('identity resolver requires a database pool');
|
||||
|
||||
return {
|
||||
async fromSessionToken(token) {
|
||||
if (!token) return null;
|
||||
const [rows] = await pool.query(
|
||||
`SELECT s.user_id, u.username
|
||||
FROM h5_login_sessions s
|
||||
JOIN h5_users u ON u.id = s.user_id
|
||||
WHERE s.token_hash = ?
|
||||
AND s.revoked_at IS NULL
|
||||
AND s.expires_at > ?
|
||||
AND u.status <> 'disabled'
|
||||
LIMIT 1`,
|
||||
[sessionTokenHash(token), now()],
|
||||
);
|
||||
const row = rows[0];
|
||||
return row
|
||||
? {
|
||||
id: String(row.user_id ?? ''),
|
||||
username: String(row.username ?? ''),
|
||||
}
|
||||
: null;
|
||||
},
|
||||
|
||||
async fromWechatOpenid(openid) {
|
||||
if (!openid || !appId) return null;
|
||||
const [rows] = await pool.query(
|
||||
`SELECT wi.user_id, u.username
|
||||
FROM h5_user_wechat_identities wi
|
||||
JOIN h5_users u ON u.id = wi.user_id
|
||||
WHERE wi.app_id = ?
|
||||
AND wi.openid = ?
|
||||
AND u.status <> 'disabled'
|
||||
LIMIT 1`,
|
||||
[appId, openid],
|
||||
);
|
||||
const row = rows[0];
|
||||
return row
|
||||
? {
|
||||
id: String(row.user_id ?? ''),
|
||||
username: String(row.username ?? ''),
|
||||
wechatUserId: String(row.username ?? ''),
|
||||
}
|
||||
: null;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export async function verifyCanaryPolicySelectors(pool, policy) {
|
||||
const selectors = [
|
||||
...[...(policy.userIds ?? [])].map((value) => ['id', value]),
|
||||
...[...(policy.usernames ?? [])].map((value) => ['username', value]),
|
||||
...[...(policy.wechatUserIds ?? [])].map((value) => ['wechat', value]),
|
||||
];
|
||||
for (const [kind, value] of selectors) {
|
||||
if (kind === 'username') {
|
||||
const [rows] = await pool.query(
|
||||
`SELECT id FROM h5_users
|
||||
WHERE LOWER(username) = LOWER(?) AND status <> 'disabled'
|
||||
LIMIT 2`,
|
||||
[value],
|
||||
);
|
||||
if (rows.length !== 1) {
|
||||
throw new Error('configured canary username does not resolve uniquely');
|
||||
}
|
||||
continue;
|
||||
}
|
||||
const lookupColumn = kind === 'wechat' ? 'u.username' : 'u.id';
|
||||
const [rows] = await pool.query(
|
||||
`SELECT u.id,
|
||||
EXISTS(
|
||||
SELECT 1 FROM h5_user_wechat_identities wi WHERE wi.user_id = u.id
|
||||
) AS has_wechat
|
||||
FROM h5_users u
|
||||
WHERE ${lookupColumn} = ? AND u.status <> 'disabled'
|
||||
LIMIT 2`,
|
||||
[value],
|
||||
);
|
||||
if (rows.length !== 1) {
|
||||
throw new Error('configured canary immutable user selector does not resolve uniquely');
|
||||
}
|
||||
if (kind === 'wechat' && Number(rows[0].has_wechat ?? 0) !== 1) {
|
||||
throw new Error('configured WeChat canary user has no bound immutable identity');
|
||||
}
|
||||
}
|
||||
return {
|
||||
userIds: policy.userIds?.size ?? 0,
|
||||
usernames: policy.usernames?.size ?? 0,
|
||||
wechatUserIds: policy.wechatUserIds?.size ?? 0,
|
||||
};
|
||||
}
|
||||
|
||||
function requestBody(req, limit = DEFAULT_BODY_LIMIT) {
|
||||
return new Promise((resolve, reject) => {
|
||||
const chunks = [];
|
||||
let bytes = 0;
|
||||
req.on('data', (chunk) => {
|
||||
bytes += chunk.length;
|
||||
if (bytes > limit) {
|
||||
reject(Object.assign(new Error('request body too large'), { statusCode: 413 }));
|
||||
req.destroy();
|
||||
return;
|
||||
}
|
||||
chunks.push(chunk);
|
||||
});
|
||||
req.on('end', () => resolve(Buffer.concat(chunks)));
|
||||
req.on('error', reject);
|
||||
});
|
||||
}
|
||||
|
||||
function healthRequest(target, timeoutMs, expectedRole = '') {
|
||||
return new Promise((resolve) => {
|
||||
const request = http.request(
|
||||
{
|
||||
hostname: target.hostname,
|
||||
port: target.port,
|
||||
method: 'GET',
|
||||
path: `${target.pathname}/api/status`.replace(/\/{2,}/g, '/'),
|
||||
headers: { connection: 'close' },
|
||||
},
|
||||
(response) => {
|
||||
response.resume();
|
||||
response.once('end', () => {
|
||||
const runtimeRole = String(
|
||||
response.headers['x-memind-runtime-role'] ?? '',
|
||||
).trim().toLowerCase();
|
||||
resolve(
|
||||
response.statusCode === 200
|
||||
&& (!expectedRole || runtimeRole === expectedRole),
|
||||
);
|
||||
});
|
||||
},
|
||||
);
|
||||
request.setTimeout(timeoutMs, () => request.destroy());
|
||||
request.once('error', () => resolve(false));
|
||||
request.end();
|
||||
});
|
||||
}
|
||||
|
||||
function appendForwardedFor(current, remoteAddress) {
|
||||
const values = [String(current ?? '').trim(), String(remoteAddress ?? '').trim()]
|
||||
.filter(Boolean);
|
||||
return values.join(', ');
|
||||
}
|
||||
|
||||
export function createMemindCanaryProxy({
|
||||
stableTarget,
|
||||
candidateTarget,
|
||||
policy,
|
||||
identityResolver,
|
||||
diagnosticSecret,
|
||||
candidateHealthTtlMs = 1_000,
|
||||
candidateHealthTimeoutMs = 750,
|
||||
logger = console,
|
||||
} = {}) {
|
||||
const stable = normalizeLoopbackTarget(stableTarget, 'stable target');
|
||||
const candidate = normalizeLoopbackTarget(candidateTarget, 'candidate target');
|
||||
if (!policy) throw new Error('canary proxy requires a routing policy');
|
||||
if (!identityResolver) throw new Error('canary proxy requires an identity resolver');
|
||||
if (!diagnosticSecret) throw new Error('canary proxy requires a diagnostic secret');
|
||||
|
||||
let healthCache = { checkedAt: 0, healthy: false };
|
||||
let healthPromise = null;
|
||||
|
||||
const candidateHealthy = async ({ force = false } = {}) => {
|
||||
const currentTime = Date.now();
|
||||
if (!force && currentTime - healthCache.checkedAt < candidateHealthTtlMs) {
|
||||
return healthCache.healthy;
|
||||
}
|
||||
if (!healthPromise) {
|
||||
healthPromise = healthRequest(candidate, candidateHealthTimeoutMs, 'candidate')
|
||||
.then((healthy) => {
|
||||
healthCache = { checkedAt: Date.now(), healthy };
|
||||
return healthy;
|
||||
})
|
||||
.finally(() => {
|
||||
healthPromise = null;
|
||||
});
|
||||
}
|
||||
return healthPromise;
|
||||
};
|
||||
|
||||
const stableHealthy = () => healthRequest(stable, candidateHealthTimeoutMs);
|
||||
|
||||
const resolveIdentity = async (req, bufferedBody) => {
|
||||
if (
|
||||
req.method === 'POST'
|
||||
&& String(req.url ?? '').split('?')[0] === '/webhooks/wechat-mp/messages'
|
||||
) {
|
||||
return identityResolver.fromWechatOpenid(
|
||||
parseWechatOpenid(bufferedBody?.toString('utf8')),
|
||||
);
|
||||
}
|
||||
const token = parseCookieHeader(req.headers.cookie).get(SESSION_COOKIE);
|
||||
return identityResolver.fromSessionToken(token);
|
||||
};
|
||||
|
||||
const selectTarget = async (req, bufferedBody) => {
|
||||
let identity = null;
|
||||
try {
|
||||
identity = await resolveIdentity(req, bufferedBody);
|
||||
} catch (error) {
|
||||
logger.warn?.(
|
||||
'Canary identity resolution failed closed to stable:',
|
||||
error instanceof Error ? error.message : String(error),
|
||||
);
|
||||
return { route: 'stable', target: stable };
|
||||
}
|
||||
if (resolveCanaryTarget(identity, policy) !== 'candidate') {
|
||||
return { route: 'stable', target: stable };
|
||||
}
|
||||
if (!(await candidateHealthy())) {
|
||||
return { route: 'stable', target: stable };
|
||||
}
|
||||
return { route: 'candidate', target: candidate };
|
||||
};
|
||||
|
||||
const proxyRequest = (req, res, selection, bufferedBody, allowFallback = true) => {
|
||||
const headers = {
|
||||
...req.headers,
|
||||
'x-forwarded-for': appendForwardedFor(
|
||||
req.headers['x-forwarded-for'],
|
||||
req.socket.remoteAddress,
|
||||
),
|
||||
'x-memind-canary-route': selection.route,
|
||||
};
|
||||
delete headers['x-memind-runtime-route'];
|
||||
if (bufferedBody) {
|
||||
headers['content-length'] = String(bufferedBody.length);
|
||||
delete headers['transfer-encoding'];
|
||||
}
|
||||
const hasDeclaredBody =
|
||||
Boolean(req.headers['content-length'])
|
||||
|| Boolean(req.headers['transfer-encoding']);
|
||||
const safeToReplay =
|
||||
Boolean(bufferedBody)
|
||||
|| ['GET', 'HEAD', 'OPTIONS'].includes(String(req.method ?? '').toUpperCase())
|
||||
|| !hasDeclaredBody;
|
||||
|
||||
let connected = false;
|
||||
const upstream = http.request(
|
||||
{
|
||||
hostname: selection.target.hostname,
|
||||
port: selection.target.port,
|
||||
method: req.method,
|
||||
path: `${selection.target.pathname}${req.url ?? '/'}`.replace(/\/{2,}/g, '/'),
|
||||
headers,
|
||||
},
|
||||
(upstreamResponse) => {
|
||||
const responseHeaders = {
|
||||
...upstreamResponse.headers,
|
||||
'x-memind-runtime-route': selection.route,
|
||||
};
|
||||
res.writeHead(upstreamResponse.statusCode ?? 502, responseHeaders);
|
||||
upstreamResponse.pipe(res);
|
||||
},
|
||||
);
|
||||
upstream.on('socket', (socket) => {
|
||||
socket.once('connect', () => {
|
||||
connected = true;
|
||||
});
|
||||
});
|
||||
upstream.once('error', (error) => {
|
||||
if (
|
||||
allowFallback
|
||||
&& selection.route === 'candidate'
|
||||
&& safeToReplay
|
||||
&& !connected
|
||||
&& RETRYABLE_CONNECT_ERRORS.has(error?.code)
|
||||
&& !res.headersSent
|
||||
) {
|
||||
healthCache = { checkedAt: Date.now(), healthy: false };
|
||||
proxyRequest(
|
||||
req,
|
||||
res,
|
||||
{ route: 'stable', target: stable },
|
||||
bufferedBody,
|
||||
false,
|
||||
);
|
||||
return;
|
||||
}
|
||||
if (!res.headersSent) {
|
||||
res.writeHead(502, { 'content-type': 'application/json; charset=utf-8' });
|
||||
}
|
||||
res.end(JSON.stringify({ ok: false, message: 'canary upstream unavailable' }));
|
||||
});
|
||||
if (bufferedBody) {
|
||||
upstream.end(bufferedBody);
|
||||
} else if (!hasDeclaredBody) {
|
||||
upstream.end();
|
||||
} else {
|
||||
req.pipe(upstream);
|
||||
}
|
||||
};
|
||||
|
||||
const server = http.createServer(async (req, res) => {
|
||||
try {
|
||||
const requestUrl = new URL(req.url ?? '/', 'http://127.0.0.1');
|
||||
const pathname = requestUrl.pathname;
|
||||
if (pathname === '/__memind_canary/health') {
|
||||
if (req.headers['x-memind-canary-secret'] !== diagnosticSecret) {
|
||||
res.writeHead(404).end();
|
||||
return;
|
||||
}
|
||||
const [stableOk, candidateOk] = await Promise.all([
|
||||
stableHealthy(),
|
||||
candidateHealthy({ force: true }),
|
||||
]);
|
||||
res.writeHead(stableOk ? 200 : 503, {
|
||||
'content-type': 'application/json; charset=utf-8',
|
||||
'cache-control': 'no-store',
|
||||
});
|
||||
res.end(JSON.stringify({
|
||||
ok: stableOk,
|
||||
stable: stableOk,
|
||||
candidate: candidateOk,
|
||||
policy: {
|
||||
userIds: policy.userIds?.size ?? 0,
|
||||
usernames: policy.usernames?.size ?? 0,
|
||||
wechatUserIds: policy.wechatUserIds?.size ?? 0,
|
||||
},
|
||||
}));
|
||||
return;
|
||||
}
|
||||
if (pathname === '/__memind_canary/route-probe') {
|
||||
if (req.headers['x-memind-canary-secret'] !== diagnosticSecret) {
|
||||
res.writeHead(404).end();
|
||||
return;
|
||||
}
|
||||
const identity = {
|
||||
id: requestUrl.searchParams.get('id') ?? '',
|
||||
username: requestUrl.searchParams.get('username') ?? '',
|
||||
wechatUserId: requestUrl.searchParams.get('wechat_user_id') ?? '',
|
||||
};
|
||||
const configuredTarget = resolveCanaryTarget(identity, policy);
|
||||
const candidateOk =
|
||||
configuredTarget === 'candidate' ? await candidateHealthy({ force: true }) : false;
|
||||
const route =
|
||||
configuredTarget === 'candidate' && candidateOk ? 'candidate' : 'stable';
|
||||
res.writeHead(200, {
|
||||
'content-type': 'application/json; charset=utf-8',
|
||||
'cache-control': 'no-store',
|
||||
});
|
||||
res.end(JSON.stringify({
|
||||
route,
|
||||
configuredTarget,
|
||||
candidate: candidateOk,
|
||||
}));
|
||||
return;
|
||||
}
|
||||
|
||||
const needsBody =
|
||||
req.method === 'POST'
|
||||
&& pathname === '/webhooks/wechat-mp/messages';
|
||||
const bufferedBody = needsBody ? await requestBody(req) : null;
|
||||
const selection = await selectTarget(req, bufferedBody);
|
||||
proxyRequest(req, res, selection, bufferedBody);
|
||||
} catch (error) {
|
||||
const statusCode = Number(error?.statusCode ?? 500);
|
||||
if (!res.headersSent) {
|
||||
res.writeHead(statusCode, { 'content-type': 'application/json; charset=utf-8' });
|
||||
}
|
||||
res.end(JSON.stringify({ ok: false, message: 'canary routing failed closed' }));
|
||||
}
|
||||
});
|
||||
|
||||
server.on('upgrade', async (req, socket, head) => {
|
||||
let selection;
|
||||
try {
|
||||
selection = await selectTarget(req, null);
|
||||
} catch {
|
||||
selection = { route: 'stable', target: stable };
|
||||
}
|
||||
const headers = {
|
||||
...req.headers,
|
||||
connection: 'Upgrade',
|
||||
upgrade: req.headers.upgrade ?? 'websocket',
|
||||
'x-memind-canary-route': selection.route,
|
||||
};
|
||||
const upstream = http.request({
|
||||
hostname: selection.target.hostname,
|
||||
port: selection.target.port,
|
||||
method: req.method,
|
||||
path: req.url,
|
||||
headers,
|
||||
});
|
||||
upstream.on('upgrade', (response, upstreamSocket, upstreamHead) => {
|
||||
const statusLine =
|
||||
`HTTP/1.1 ${response.statusCode ?? 101} ${response.statusMessage ?? 'Switching Protocols'}\r\n`;
|
||||
const responseHeaders = Object.entries(response.headers)
|
||||
.flatMap(([name, value]) => {
|
||||
const values = Array.isArray(value) ? value : [value];
|
||||
return values.filter((item) => item != null).map((item) => `${name}: ${item}\r\n`);
|
||||
})
|
||||
.join('');
|
||||
socket.write(`${statusLine}${responseHeaders}\r\n`);
|
||||
if (upstreamHead.length) socket.write(upstreamHead);
|
||||
if (head.length) upstreamSocket.write(head);
|
||||
upstreamSocket.pipe(socket);
|
||||
socket.pipe(upstreamSocket);
|
||||
});
|
||||
upstream.once('error', () => socket.destroy());
|
||||
upstream.end();
|
||||
});
|
||||
|
||||
return {
|
||||
server,
|
||||
candidateHealthy,
|
||||
stableHealthy,
|
||||
};
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const persistentRoot =
|
||||
process.env.MEMIND_PORTAL_H5_ROOT
|
||||
|| process.env.MEMIND_CANARY_STABLE_ROOT
|
||||
|| process.cwd();
|
||||
loadMemindEnvFiles(persistentRoot);
|
||||
const policy = canaryPolicyFromEnv(process.env);
|
||||
const pool = createIdentityPool(process.env);
|
||||
const selectorSummary = await verifyCanaryPolicySelectors(pool, policy);
|
||||
if (process.argv.includes('--check')) {
|
||||
console.log(JSON.stringify({ ok: true, policy: selectorSummary }));
|
||||
await pool.end();
|
||||
return;
|
||||
}
|
||||
|
||||
const identityResolver = createCanaryIdentityResolver({
|
||||
pool,
|
||||
appId:
|
||||
process.env.H5_WECHAT_MP_APP_ID
|
||||
|| process.env.H5_WECHAT_APP_ID
|
||||
|| '',
|
||||
});
|
||||
const listenHost = process.env.MEMIND_CANARY_PROXY_HOST || '127.0.0.1';
|
||||
const listenPort = Number(process.env.MEMIND_CANARY_PROXY_PORT ?? 18080);
|
||||
const proxy = createMemindCanaryProxy({
|
||||
stableTarget: process.env.MEMIND_CANARY_STABLE_URL || 'http://127.0.0.1:8081',
|
||||
candidateTarget:
|
||||
process.env.MEMIND_CANARY_CANDIDATE_URL || 'http://127.0.0.1:18081',
|
||||
policy,
|
||||
identityResolver,
|
||||
diagnosticSecret: process.env.MEMIND_CANARY_DIAGNOSTIC_SECRET,
|
||||
});
|
||||
|
||||
const shutdown = async () => {
|
||||
proxy.server.close();
|
||||
await pool.end().catch(() => {});
|
||||
};
|
||||
process.once('SIGTERM', shutdown);
|
||||
process.once('SIGINT', shutdown);
|
||||
proxy.server.listen(listenPort, listenHost, () => {
|
||||
console.log(
|
||||
`Memind canary proxy listening on http://${listenHost}:${listenPort} `
|
||||
+ `(stable=8081 candidate=18081 selectors=${JSON.stringify(selectorSummary)})`,
|
||||
);
|
||||
});
|
||||
}
|
||||
|
||||
if (
|
||||
process.argv[1]
|
||||
&& import.meta.url === pathToFileURL(process.argv[1]).href
|
||||
) {
|
||||
main().catch((error) => {
|
||||
console.error(error instanceof Error ? error.message : String(error));
|
||||
process.exit(1);
|
||||
});
|
||||
}
|
||||
@@ -0,0 +1,397 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import http from 'node:http';
|
||||
import test from 'node:test';
|
||||
|
||||
import { createCanaryPolicy } from './release-gate/canary-routing.mjs';
|
||||
import {
|
||||
createCanaryIdentityResolver,
|
||||
createMemindCanaryProxy,
|
||||
parseCookieHeader,
|
||||
parseWechatOpenid,
|
||||
verifyCanaryPolicySelectors,
|
||||
} from './memind-canary-proxy.mjs';
|
||||
|
||||
function listen(server) {
|
||||
return new Promise((resolve, reject) => {
|
||||
server.once('error', reject);
|
||||
server.listen(0, '127.0.0.1', () => {
|
||||
const address = server.address();
|
||||
resolve(`http://127.0.0.1:${address.port}`);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
function close(server) {
|
||||
return new Promise((resolve) => server.close(resolve));
|
||||
}
|
||||
|
||||
function request(baseUrl, {
|
||||
method = 'GET',
|
||||
path = '/',
|
||||
headers = {},
|
||||
body = '',
|
||||
} = {}) {
|
||||
const target = new URL(path, baseUrl);
|
||||
return new Promise((resolve, reject) => {
|
||||
const req = http.request(
|
||||
{
|
||||
hostname: target.hostname,
|
||||
port: target.port,
|
||||
path: `${target.pathname}${target.search}`,
|
||||
method,
|
||||
headers: {
|
||||
...headers,
|
||||
...(body ? { 'content-length': Buffer.byteLength(body) } : {}),
|
||||
},
|
||||
},
|
||||
(res) => {
|
||||
const chunks = [];
|
||||
res.on('data', (chunk) => chunks.push(chunk));
|
||||
res.on('end', () => resolve({
|
||||
status: res.statusCode,
|
||||
headers: res.headers,
|
||||
body: Buffer.concat(chunks).toString('utf8'),
|
||||
}));
|
||||
},
|
||||
);
|
||||
req.once('error', reject);
|
||||
req.end(body);
|
||||
});
|
||||
}
|
||||
|
||||
function upstream(name) {
|
||||
return http.createServer((req, res) => {
|
||||
if (req.url === '/api/status') {
|
||||
res.writeHead(200, { 'x-memind-runtime-role': name });
|
||||
res.end('ok');
|
||||
return;
|
||||
}
|
||||
const chunks = [];
|
||||
req.on('data', (chunk) => chunks.push(chunk));
|
||||
req.on('end', () => {
|
||||
res.writeHead(200, { 'content-type': 'text/plain' });
|
||||
res.end(`${name}:${Buffer.concat(chunks).toString('utf8')}`);
|
||||
});
|
||||
});
|
||||
}
|
||||
|
||||
function candidateThatStopsAfterHealth() {
|
||||
const server = http.createServer((req, res) => {
|
||||
if (req.url !== '/api/status') {
|
||||
res.writeHead(200);
|
||||
res.end('candidate');
|
||||
return;
|
||||
}
|
||||
server.close();
|
||||
res.writeHead(200, { 'x-memind-runtime-role': 'candidate' });
|
||||
res.end('ok');
|
||||
});
|
||||
return server;
|
||||
}
|
||||
|
||||
test('cookie and WeChat parsers preserve immutable routing inputs', () => {
|
||||
const cookies = parseCookieHeader('a=1; tkmind_user_session=abc%2D123');
|
||||
assert.equal(cookies.get('tkmind_user_session'), 'abc-123');
|
||||
assert.equal(
|
||||
parseWechatOpenid(
|
||||
'<xml><FromUserName><![CDATA[openid-target]]></FromUserName></xml>',
|
||||
),
|
||||
'openid-target',
|
||||
);
|
||||
assert.equal(parseWechatOpenid('<xml></xml>'), '');
|
||||
});
|
||||
|
||||
test('identity resolver hashes sessions and maps WeChat openid to bound user id', async () => {
|
||||
const calls = [];
|
||||
const resolver = createCanaryIdentityResolver({
|
||||
pool: {
|
||||
async query(sql, params) {
|
||||
calls.push({ sql, params });
|
||||
if (sql.includes('h5_login_sessions')) {
|
||||
return [[{ user_id: 'user-john', username: 'john' }]];
|
||||
}
|
||||
return [[{ user_id: 'wx_ul610et8', username: 'wechat_user' }]];
|
||||
},
|
||||
},
|
||||
appId: 'app-1',
|
||||
now: () => 123,
|
||||
});
|
||||
|
||||
assert.deepEqual(await resolver.fromSessionToken('token-1'), {
|
||||
id: 'user-john',
|
||||
username: 'john',
|
||||
});
|
||||
assert.match(calls[0].params[0], /^[a-f0-9]{64}$/);
|
||||
assert.equal(calls[0].params[1], 123);
|
||||
|
||||
assert.deepEqual(await resolver.fromWechatOpenid('openid-1'), {
|
||||
id: 'wx_ul610et8',
|
||||
username: 'wechat_user',
|
||||
wechatUserId: 'wechat_user',
|
||||
});
|
||||
assert.deepEqual(calls[1].params, ['app-1', 'openid-1']);
|
||||
});
|
||||
|
||||
test('WeChat canary selector validates the bound unique username, not a production UUID', async () => {
|
||||
const calls = [];
|
||||
const summary = await verifyCanaryPolicySelectors(
|
||||
{
|
||||
async query(sql, params) {
|
||||
calls.push({ sql, params });
|
||||
if (sql.includes('LOWER(username)')) return [[{ id: 'john-uuid' }]];
|
||||
return [[{ id: 'wechat-uuid', has_wechat: 1 }]];
|
||||
},
|
||||
},
|
||||
createCanaryPolicy({
|
||||
usernames: ['john'],
|
||||
wechatUserIds: ['wx_ul610et8'],
|
||||
}),
|
||||
);
|
||||
|
||||
assert.deepEqual(summary, { userIds: 0, usernames: 1, wechatUserIds: 1 });
|
||||
assert.match(calls[1].sql, /WHERE u\.username = \?/);
|
||||
assert.deepEqual(calls[1].params, ['wx_ul610et8']);
|
||||
});
|
||||
|
||||
test('proxy routes only immutable canary identities and preserves WeChat bodies', async (t) => {
|
||||
const stableServer = upstream('stable');
|
||||
const candidateServer = upstream('candidate');
|
||||
const stableUrl = await listen(stableServer);
|
||||
const candidateUrl = await listen(candidateServer);
|
||||
const policy = createCanaryPolicy({
|
||||
usernames: ['john'],
|
||||
wechatUserIds: ['wx_ul610et8'],
|
||||
});
|
||||
const identityResolver = {
|
||||
async fromSessionToken(token) {
|
||||
if (token === 'john-token') return { id: 'user-john', username: 'john' };
|
||||
if (token === 'same-name-token') {
|
||||
return { id: 'other-user', username: 'other', displayName: '唐' };
|
||||
}
|
||||
return null;
|
||||
},
|
||||
async fromWechatOpenid(openid) {
|
||||
if (openid === 'openid-target') {
|
||||
return {
|
||||
id: 'wechat-user-uuid',
|
||||
username: 'wx_ul610et8',
|
||||
wechatUserId: 'wx_ul610et8',
|
||||
};
|
||||
}
|
||||
return null;
|
||||
},
|
||||
};
|
||||
const proxy = createMemindCanaryProxy({
|
||||
stableTarget: stableUrl,
|
||||
candidateTarget: candidateUrl,
|
||||
policy,
|
||||
identityResolver,
|
||||
diagnosticSecret: 'diagnostic-secret',
|
||||
candidateHealthTtlMs: 0,
|
||||
});
|
||||
const proxyUrl = await listen(proxy.server);
|
||||
t.after(async () => {
|
||||
await close(proxy.server);
|
||||
await close(stableServer);
|
||||
if (candidateServer.listening) await close(candidateServer);
|
||||
});
|
||||
|
||||
const john = await request(proxyUrl, {
|
||||
headers: { cookie: 'tkmind_user_session=john-token' },
|
||||
});
|
||||
assert.equal(john.body, 'candidate:');
|
||||
assert.equal(john.headers['x-memind-runtime-route'], 'candidate');
|
||||
|
||||
const missingIdentity = await request(proxyUrl);
|
||||
assert.equal(missingIdentity.body, 'stable:');
|
||||
assert.equal(missingIdentity.headers['x-memind-runtime-route'], 'stable');
|
||||
|
||||
const sameNickname = await request(proxyUrl, {
|
||||
headers: { cookie: 'tkmind_user_session=same-name-token' },
|
||||
});
|
||||
assert.equal(sameNickname.body, 'stable:');
|
||||
|
||||
const xml =
|
||||
'<xml><FromUserName><![CDATA[openid-target]]></FromUserName>'
|
||||
+ '<Content><![CDATA[hello]]></Content></xml>';
|
||||
const wechat = await request(proxyUrl, {
|
||||
method: 'POST',
|
||||
path: '/webhooks/wechat-mp/messages',
|
||||
headers: { 'content-type': 'text/xml' },
|
||||
body: xml,
|
||||
});
|
||||
assert.equal(wechat.body, `candidate:${xml}`);
|
||||
assert.equal(wechat.headers['x-memind-runtime-route'], 'candidate');
|
||||
|
||||
const health = await request(proxyUrl, {
|
||||
path: '/__memind_canary/health',
|
||||
headers: { 'x-memind-canary-secret': 'diagnostic-secret' },
|
||||
});
|
||||
assert.equal(health.status, 200);
|
||||
assert.deepEqual(JSON.parse(health.body), {
|
||||
ok: true,
|
||||
stable: true,
|
||||
candidate: true,
|
||||
policy: { userIds: 0, usernames: 1, wechatUserIds: 1 },
|
||||
});
|
||||
|
||||
await close(candidateServer);
|
||||
const routeProbe = await request(proxyUrl, {
|
||||
path: '/__memind_canary/route-probe?username=john',
|
||||
headers: { 'x-memind-canary-secret': 'diagnostic-secret' },
|
||||
});
|
||||
assert.deepEqual(JSON.parse(routeProbe.body), {
|
||||
route: 'stable',
|
||||
configuredTarget: 'candidate',
|
||||
candidate: false,
|
||||
});
|
||||
const fallback = await request(proxyUrl, {
|
||||
headers: { cookie: 'tkmind_user_session=john-token' },
|
||||
});
|
||||
assert.equal(fallback.body, 'stable:');
|
||||
assert.equal(fallback.headers['x-memind-runtime-route'], 'stable');
|
||||
});
|
||||
|
||||
test('identity lookup errors fail closed to stable', async (t) => {
|
||||
const stableServer = upstream('stable');
|
||||
const candidateServer = upstream('candidate');
|
||||
const stableUrl = await listen(stableServer);
|
||||
const candidateUrl = await listen(candidateServer);
|
||||
const proxy = createMemindCanaryProxy({
|
||||
stableTarget: stableUrl,
|
||||
candidateTarget: candidateUrl,
|
||||
policy: createCanaryPolicy({ usernames: ['john'] }),
|
||||
identityResolver: {
|
||||
async fromSessionToken() {
|
||||
throw new Error('database unavailable');
|
||||
},
|
||||
async fromWechatOpenid() {
|
||||
throw new Error('database unavailable');
|
||||
},
|
||||
},
|
||||
diagnosticSecret: 'diagnostic-secret',
|
||||
candidateHealthTtlMs: 0,
|
||||
logger: { warn() {} },
|
||||
});
|
||||
const proxyUrl = await listen(proxy.server);
|
||||
t.after(async () => {
|
||||
await close(proxy.server);
|
||||
await close(stableServer);
|
||||
await close(candidateServer);
|
||||
});
|
||||
|
||||
const response = await request(proxyUrl, {
|
||||
headers: { cookie: 'tkmind_user_session=john-token' },
|
||||
});
|
||||
assert.equal(response.body, 'stable:');
|
||||
assert.equal(response.headers['x-memind-runtime-route'], 'stable');
|
||||
});
|
||||
|
||||
test('candidate health requires the explicit candidate runtime role', async (t) => {
|
||||
const stableServer = upstream('stable');
|
||||
const wrongRoleServer = upstream('stable');
|
||||
const stableUrl = await listen(stableServer);
|
||||
const wrongRoleUrl = await listen(wrongRoleServer);
|
||||
const proxy = createMemindCanaryProxy({
|
||||
stableTarget: stableUrl,
|
||||
candidateTarget: wrongRoleUrl,
|
||||
policy: createCanaryPolicy({ usernames: ['john'] }),
|
||||
identityResolver: {
|
||||
async fromSessionToken() {
|
||||
return { id: 'user-john', username: 'john' };
|
||||
},
|
||||
async fromWechatOpenid() {
|
||||
return null;
|
||||
},
|
||||
},
|
||||
diagnosticSecret: 'diagnostic-secret',
|
||||
candidateHealthTtlMs: 0,
|
||||
});
|
||||
const proxyUrl = await listen(proxy.server);
|
||||
t.after(async () => {
|
||||
await close(proxy.server);
|
||||
await close(stableServer);
|
||||
await close(wrongRoleServer);
|
||||
});
|
||||
|
||||
const response = await request(proxyUrl, {
|
||||
headers: { cookie: 'tkmind_user_session=john-token' },
|
||||
});
|
||||
assert.equal(response.body, 'stable:');
|
||||
assert.equal(response.headers['x-memind-runtime-route'], 'stable');
|
||||
});
|
||||
|
||||
test('candidate connect race retries only replay-safe requests on stable', async (t) => {
|
||||
const stableServer = upstream('stable');
|
||||
const candidateServer = candidateThatStopsAfterHealth();
|
||||
const stableUrl = await listen(stableServer);
|
||||
const candidateUrl = await listen(candidateServer);
|
||||
const proxy = createMemindCanaryProxy({
|
||||
stableTarget: stableUrl,
|
||||
candidateTarget: candidateUrl,
|
||||
policy: createCanaryPolicy({ usernames: ['john'] }),
|
||||
identityResolver: {
|
||||
async fromSessionToken() {
|
||||
return { id: 'user-john', username: 'john' };
|
||||
},
|
||||
async fromWechatOpenid() {
|
||||
return null;
|
||||
},
|
||||
},
|
||||
diagnosticSecret: 'diagnostic-secret',
|
||||
candidateHealthTtlMs: 0,
|
||||
});
|
||||
const proxyUrl = await listen(proxy.server);
|
||||
t.after(async () => {
|
||||
await close(proxy.server);
|
||||
await close(stableServer);
|
||||
if (candidateServer.listening) await close(candidateServer);
|
||||
});
|
||||
|
||||
const response = await request(proxyUrl, {
|
||||
headers: { cookie: 'tkmind_user_session=john-token' },
|
||||
});
|
||||
assert.equal(response.status, 200);
|
||||
assert.equal(response.body, 'stable:');
|
||||
assert.equal(response.headers['x-memind-runtime-route'], 'stable');
|
||||
});
|
||||
|
||||
test('candidate connect race never replays an unbuffered write to stable', async (t) => {
|
||||
const stableServer = upstream('stable');
|
||||
const candidateServer = candidateThatStopsAfterHealth();
|
||||
const stableUrl = await listen(stableServer);
|
||||
const candidateUrl = await listen(candidateServer);
|
||||
const proxy = createMemindCanaryProxy({
|
||||
stableTarget: stableUrl,
|
||||
candidateTarget: candidateUrl,
|
||||
policy: createCanaryPolicy({ usernames: ['john'] }),
|
||||
identityResolver: {
|
||||
async fromSessionToken() {
|
||||
return { id: 'user-john', username: 'john' };
|
||||
},
|
||||
async fromWechatOpenid() {
|
||||
return null;
|
||||
},
|
||||
},
|
||||
diagnosticSecret: 'diagnostic-secret',
|
||||
candidateHealthTtlMs: 0,
|
||||
});
|
||||
const proxyUrl = await listen(proxy.server);
|
||||
t.after(async () => {
|
||||
await close(proxy.server);
|
||||
await close(stableServer);
|
||||
if (candidateServer.listening) await close(candidateServer);
|
||||
});
|
||||
|
||||
const response = await request(proxyUrl, {
|
||||
method: 'POST',
|
||||
path: '/api/write',
|
||||
headers: {
|
||||
cookie: 'tkmind_user_session=john-token',
|
||||
'content-type': 'application/json',
|
||||
},
|
||||
body: '{"value":1}',
|
||||
});
|
||||
assert.equal(response.status, 502);
|
||||
assert.match(response.body, /canary upstream unavailable/);
|
||||
});
|
||||
@@ -39,6 +39,14 @@
|
||||
"build": "vite build",
|
||||
"build:deep-search-runtime": "node scripts/build-deep-search-runtime.mjs",
|
||||
"build:portal-runtime": "node scripts/build-portal-runtime.mjs",
|
||||
"test:release-gate:unit": "node --test release-gate/*.test.mjs memind-canary-proxy.test.mjs server/portal-runtime-role.test.mjs",
|
||||
"test:release-gate:deterministic": "node scripts/run-release-gate.mjs --mode deterministic",
|
||||
"test:release-gate:scenarios": "node scripts/run-release-gate.mjs --mode scenarios",
|
||||
"test:release-gate:browser": "node scripts/run-release-gate.mjs --mode browser",
|
||||
"test:release-gate:providers": "node scripts/run-release-gate.mjs --mode providers",
|
||||
"test:release-gate:upgrade": "node scripts/run-release-gate.mjs --mode upgrade",
|
||||
"test:release-gate:all": "node scripts/run-release-gate.mjs --mode all",
|
||||
"verify:release-gate-report": "node scripts/verify-release-gate-report.mjs",
|
||||
"build:imgproxy-runtime": "bash scripts/build-imgproxy-runtime-image.sh",
|
||||
"build:mindspace-service-runtime": "node scripts/build-mindspace-service-runtime.mjs",
|
||||
"check:mindspace-public-links": "node scripts/check-mindspace-public-links.mjs --downloads-only",
|
||||
|
||||
@@ -5,6 +5,7 @@ import path from 'node:path';
|
||||
|
||||
export const REQUIRED_PORTAL_RUNTIME_PATHS = Object.freeze([
|
||||
'server.mjs',
|
||||
'memind-canary-proxy.mjs',
|
||||
'wechat-mp.bundle.mjs',
|
||||
'mindspace-sandbox-mcp.mjs',
|
||||
'tkmind-search-mcp.mjs',
|
||||
@@ -13,6 +14,9 @@ export const REQUIRED_PORTAL_RUNTIME_PATHS = Object.freeze([
|
||||
'dist',
|
||||
'package.json',
|
||||
'scripts/run-memind-portal-prod.sh',
|
||||
'scripts/run-memind-portal-candidate.sh',
|
||||
'scripts/run-memind-canary-proxy-prod.sh',
|
||||
'scripts/goosed-canary.compose.yml',
|
||||
'scripts/check-mindspace-public-links.mjs',
|
||||
'scripts/load-env.mjs',
|
||||
'scripts/wechat-mp-menu.mjs',
|
||||
|
||||
@@ -979,12 +979,14 @@ export const AUTOMATION_SUITES = Object.freeze([
|
||||
cases: {
|
||||
'REL-10': [
|
||||
'release contract: block new Agent runs, drain active work within a bound, checksum and back up, then recheck zero active runs immediately before swap',
|
||||
'canary contract: immutable user and WeChat identities route to an isolated candidate, while missing identity, resolver errors, and candidate failure fall back to stable',
|
||||
],
|
||||
},
|
||||
command: [
|
||||
process.execPath,
|
||||
'--test',
|
||||
'release-gate/canary-routing.test.mjs',
|
||||
'memind-canary-proxy.test.mjs',
|
||||
'release-gate/release-runtime-safety.test.mjs',
|
||||
],
|
||||
},
|
||||
|
||||
@@ -5,6 +5,7 @@ import path from 'node:path';
|
||||
import test from 'node:test';
|
||||
|
||||
const ROOT = path.resolve(new URL('..', import.meta.url).pathname);
|
||||
const CANARY_RELEASE = path.join(ROOT, 'scripts', 'release-portal-canary-prod.sh');
|
||||
|
||||
test('production release verifies gate report before any 103 connection', async () => {
|
||||
const source = await fs.readFile(
|
||||
@@ -77,3 +78,51 @@ test('production release rejects --skip-tests before repository or network prefl
|
||||
assert.match(result.stderr, /禁止 --skip-tests/);
|
||||
assert.doesNotMatch(`${result.stdout}\n${result.stderr}`, /103 只读预检/);
|
||||
});
|
||||
|
||||
test('production canary verifies the exact Gate artifact before any 103 preflight or upload', async () => {
|
||||
const source = await fs.readFile(CANARY_RELEASE, 'utf8');
|
||||
const gateIndex = source.indexOf('verify-release-gate-report.mjs');
|
||||
const preflightIndex = source.indexOf('Run 103 read-only preflight');
|
||||
const uploadIndex = source.indexOf('Upload the verified candidate bundle');
|
||||
assert.ok(gateIndex > 0, 'missing canary Gate verifier');
|
||||
assert.ok(preflightIndex > gateIndex, '103 preflight must follow Gate verification');
|
||||
assert.ok(uploadIndex > preflightIndex, 'upload must follow read-only preflight');
|
||||
assert.match(source, /branch.*!= "main"/);
|
||||
assert.match(source, /rev-parse origin\/main/);
|
||||
assert.match(source, /Production canary release forbids/);
|
||||
});
|
||||
|
||||
test('production canary keeps stable 8081 live and switches only after verified backups and fallback', async () => {
|
||||
const source = await fs.readFile(CANARY_RELEASE, 'utf8');
|
||||
const fullBackup = source.indexOf('Create and verify the full stable backup');
|
||||
const persistBackup = source.indexOf('Create and verify the persisted-data backup');
|
||||
const goosedStart = source.indexOf('Start an isolated goosed candidate on 18015');
|
||||
const candidateStart = source.indexOf('Start the passive candidate Portal on 18081');
|
||||
const proxyStart = source.indexOf('Start the fail-closed identity router on 18080');
|
||||
const tunnelSwitch = source.indexOf('Switch only the reverse tunnel');
|
||||
const fallbackProbe = source.indexOf('route-probe?username=john');
|
||||
assert.ok(fullBackup > 0);
|
||||
assert.ok(persistBackup > fullBackup);
|
||||
assert.ok(goosedStart > persistBackup);
|
||||
assert.ok(candidateStart > goosedStart);
|
||||
assert.ok(proxyStart > candidateStart);
|
||||
assert.ok(tunnelSwitch > proxyStart);
|
||||
assert.ok(fallbackProbe > tunnelSwitch);
|
||||
assert.match(source, /write_tunnel_plist 8081/);
|
||||
assert.match(source, /CANARY_USERNAMES="john"/);
|
||||
assert.match(source, /CANARY_WECHAT_USER_IDS="wx_ul610et8"/);
|
||||
assert.match(source, /route-probe\?\$\{probe_query\}/);
|
||||
assert.match(source, /candidate_healthy/);
|
||||
assert.match(source, /MEMIND_CANARY_RELEASE_ID/);
|
||||
assert.doesNotMatch(source, /lsof -tiTCP:8081/);
|
||||
assert.doesNotMatch(source, /bootout.*cn\.tkmind\.memind-portal/);
|
||||
});
|
||||
|
||||
test('release readiness ignores only the generated canary artifact cover path', async () => {
|
||||
const source = await fs.readFile(
|
||||
path.join(ROOT, 'scripts', 'check-release-ready.sh'),
|
||||
'utf8',
|
||||
);
|
||||
assert.match(source, /:\(exclude\)\.runtime\/portal\/public\/plaza-covers\/\*\*/);
|
||||
assert.doesNotMatch(source, /:\(exclude\)\.runtime\/\*\*/);
|
||||
});
|
||||
|
||||
@@ -136,6 +136,26 @@ async function bundleServer() {
|
||||
await run(esbuildBin, args);
|
||||
}
|
||||
|
||||
async function bundleCanaryProxy() {
|
||||
if (!(await exists(esbuildBin))) {
|
||||
throw new Error(`未找到 esbuild: ${esbuildBin}`);
|
||||
}
|
||||
console.log('==> 打包用户级灰度路由器');
|
||||
const args = [
|
||||
'memind-canary-proxy.mjs',
|
||||
'--bundle',
|
||||
'--platform=node',
|
||||
'--format=esm',
|
||||
`--target=${runtimeNodeTarget}`,
|
||||
'--outfile=.runtime/portal/memind-canary-proxy.mjs',
|
||||
'--banner:js=import { createRequire as __createRequire } from "node:module"; const require = __createRequire(import.meta.url);',
|
||||
];
|
||||
for (const pkg of externalPackages) {
|
||||
args.push(`--external:${pkg}`);
|
||||
}
|
||||
await run(esbuildBin, args);
|
||||
}
|
||||
|
||||
async function bundleSandboxMcp() {
|
||||
if (!(await exists(esbuildBin))) {
|
||||
throw new Error(`未找到 esbuild: ${esbuildBin}`);
|
||||
@@ -386,6 +406,21 @@ async function writeMetadata() {
|
||||
} else {
|
||||
throw new Error(`缺少 Portal 启动脚本: ${prodStartScript}`);
|
||||
}
|
||||
for (const scriptName of [
|
||||
'run-memind-portal-candidate.sh',
|
||||
'run-memind-canary-proxy-prod.sh',
|
||||
]) {
|
||||
const source = path.join(root, 'scripts', scriptName);
|
||||
if (!(await exists(source))) {
|
||||
throw new Error(`缺少灰度运行脚本: ${source}`);
|
||||
}
|
||||
await fs.copyFile(source, path.join(runtimeScriptsDir, scriptName));
|
||||
await fs.chmod(path.join(runtimeScriptsDir, scriptName), 0o755);
|
||||
}
|
||||
await fs.copyFile(
|
||||
path.join(root, 'scripts', 'goosed-canary.compose.yml'),
|
||||
path.join(runtimeScriptsDir, 'goosed-canary.compose.yml'),
|
||||
);
|
||||
|
||||
await fs.copyFile(
|
||||
path.join(root, 'scripts', 'load-env.mjs'),
|
||||
@@ -491,6 +526,7 @@ async function writeMetadata() {
|
||||
' tkmind-search-mcp.mjs (esbuild bundle; required when MindSearch is enabled)',
|
||||
' tkmind-excel-mcp.mjs (esbuild bundle; required when Excel Analyst is enabled)',
|
||||
' wechat-mp.bundle.mjs (esbuild bundle; hot-swappable WeChat MP module)',
|
||||
' memind-canary-proxy.mjs (immutable-identity stable/candidate router)',
|
||||
'',
|
||||
'Post-deploy validation (scripts/check-mindspace-public-links.mjs):',
|
||||
' Scans MindSpace/*/public/*.html for missing relative href/src/cover targets.',
|
||||
@@ -516,6 +552,7 @@ async function writeMetadata() {
|
||||
' Current public path: m.tkmind.cn -> 105 nginx -> 103 Portal :8081',
|
||||
' No separate public test domain remains; use explicit local/test overrides only when needed.',
|
||||
' Production H5 and WeChat links should use m.tkmind.cn unless a test rollout explicitly overrides H5_PUBLIC_BASE_URL.',
|
||||
' Canary layout: stable :8081, candidate :18081, identity router :18080, candidate goosed :18015.',
|
||||
'',
|
||||
'Streaming runtime operations:',
|
||||
' node scripts/check-stream-runtime.mjs',
|
||||
@@ -560,6 +597,7 @@ async function main() {
|
||||
await buildFrontend();
|
||||
await copyRuntimeAssets();
|
||||
await bundleServer();
|
||||
await bundleCanaryProxy();
|
||||
await bundleWechatMp();
|
||||
await bundleSandboxMcp();
|
||||
await bundleMindSearchMcp();
|
||||
@@ -571,6 +609,8 @@ async function main() {
|
||||
await copyMindspacePublicLinkTools();
|
||||
await writeMetadata();
|
||||
await fs.chmod(path.join(runtimeRoot, 'scripts', 'run-memind-portal-prod.sh'), 0o755);
|
||||
await fs.chmod(path.join(runtimeRoot, 'scripts', 'run-memind-portal-candidate.sh'), 0o755);
|
||||
await fs.chmod(path.join(runtimeRoot, 'scripts', 'run-memind-canary-proxy-prod.sh'), 0o755);
|
||||
await fs.chmod(path.join(runtimeRoot, 'scripts', 'wechat-mp-menu.mjs'), 0o755);
|
||||
await fs.chmod(path.join(runtimeRoot, 'scripts', 'check-stream-runtime.mjs'), 0o755);
|
||||
await fs.chmod(path.join(runtimeRoot, 'scripts', 'runtime-worker-drain.mjs'), 0o755);
|
||||
|
||||
@@ -53,9 +53,16 @@ case "${branch}" in
|
||||
;;
|
||||
esac
|
||||
|
||||
if [[ -n "$(git -C "${ROOT}" status --porcelain=v1 --untracked-files=all)" ]]; then
|
||||
# The runtime builder intentionally strips this legacy tracked content from the
|
||||
# source-free artifact. The complete Gate applies the same narrow exception.
|
||||
relevant_status="$(
|
||||
git -C "${ROOT}" status --porcelain=v1 --untracked-files=all -- \
|
||||
. \
|
||||
':(exclude).runtime/portal/public/plaza-covers/**'
|
||||
)"
|
||||
if [[ -n "${relevant_status}" ]]; then
|
||||
echo "Release check failed: worktree has uncommitted or untracked changes." >&2
|
||||
git -C "${ROOT}" status --short --untracked-files=all >&2
|
||||
printf '%s\n' "${relevant_status}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
|
||||
@@ -0,0 +1,39 @@
|
||||
services:
|
||||
goosed-canary:
|
||||
image: tkmind/goosed:prod-${GOOSED_TAG:-latest}
|
||||
container_name: goosed-prod-canary
|
||||
restart: unless-stopped
|
||||
env_file: ${GOOSED_PROD_ENV_FILE:-/Users/john/Project/goosed-prod/.env}
|
||||
environment:
|
||||
GOOSE_PORT: 18006
|
||||
GOOSE_AIDER_BIN: /usr/local/bin/aider
|
||||
GOOSE_OPENHANDS_BIN: /usr/local/bin/openhands
|
||||
GOOSE_OPENHANDS_RUNNER: host
|
||||
GOOSE_OPENHANDS_TIMEOUT_SECS: "900"
|
||||
ports:
|
||||
- "18015:18006"
|
||||
volumes:
|
||||
- ${CANDIDATE_RUNTIME_DIR:?}:/opt/portal:ro
|
||||
- ${GOOSED_RUNTIME_TOOLS_DIR:?}:/opt/tools:ro
|
||||
- ${GOOSED_RUNTIME_TOOLS_DIR:?}/bin/aider:/usr/local/bin/aider:ro
|
||||
- ${GOOSED_RUNTIME_TOOLS_DIR:?}/bin/openhands:/usr/local/bin/openhands:ro
|
||||
- ${GOOSED_RUNTIME_TOOLS_DIR:?}/openhands-data:/opt/tools/openhands-data
|
||||
- ${GOOSED_HARNESS_DIR:?}:/home/goose/.codex/harness
|
||||
- ${MINDSPACE_ROOT:?}:${MINDSPACE_ROOT:?}
|
||||
- ${MINDSPACE_STORAGE_ROOT:?}:${MINDSPACE_STORAGE_ROOT:?}
|
||||
- goosed-canary-data:/home/goose/.local/share/goose
|
||||
healthcheck:
|
||||
test: ["CMD", "curl", "-kfsS", "https://127.0.0.1:18006/status"]
|
||||
interval: 15s
|
||||
timeout: 5s
|
||||
retries: 3
|
||||
start_period: 20s
|
||||
networks:
|
||||
- goosed-net
|
||||
|
||||
volumes:
|
||||
goosed-canary-data:
|
||||
|
||||
networks:
|
||||
goosed-net:
|
||||
driver: bridge
|
||||
Executable
+575
@@ -0,0 +1,575 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
HOST="${STUDIO_HOST:-58.38.22.103}"
|
||||
REMOTE_ROOT="${STUDIO_REMOTE_ROOT:-/Users/john/Project}"
|
||||
STABLE_DIR="${REMOTE_ROOT}/Memind"
|
||||
CANDIDATE_BASE="${REMOTE_ROOT}/Memind-candidates"
|
||||
INCOMING_DIR="${REMOTE_ROOT}/incoming/memind-portal-canary"
|
||||
BACKUP_DIR="${REMOTE_ROOT}/backups/memind"
|
||||
GOOSED_DIR="${REMOTE_ROOT}/goosed-prod"
|
||||
RUNTIME_ROOT="${ROOT}/.runtime/portal"
|
||||
PORTAL_CANDIDATE_LABEL="cn.tkmind.memind-portal-candidate"
|
||||
CANARY_PROXY_LABEL="cn.tkmind.memind-canary-proxy"
|
||||
PORTAL_TUNNEL_LABEL="cn.tkmind.memind-portal-tunnel"
|
||||
MEMIND_PORTAL_TUNNEL_HOST="${MEMIND_PORTAL_TUNNEL_HOST:-ssh105-public}"
|
||||
MEMIND_PORTAL_TUNNEL_REMOTE_PORT="${MEMIND_PORTAL_TUNNEL_REMOTE_PORT:-19081}"
|
||||
CANARY_USERNAMES="john"
|
||||
CANARY_WECHAT_USER_IDS="wx_ul610et8"
|
||||
RELEASE_TS="$(date +%Y%m%d-%H%M%S)"
|
||||
FULL_SHA="$(git -C "${ROOT}" rev-parse HEAD)"
|
||||
SHORT_SHA="${FULL_SHA:0:7}"
|
||||
RELEASE_ID="${RELEASE_TS}-${SHORT_SHA}"
|
||||
TMP_DIR="$(mktemp -d "${TMPDIR:-/tmp}/memind-portal-canary-release.XXXXXX")"
|
||||
BUNDLE_PATH="${TMP_DIR}/memind-portal-canary-${RELEASE_ID}.tar.gz"
|
||||
MANIFEST_PATH="${TMP_DIR}/memind-portal-canary-${RELEASE_ID}.manifest.txt"
|
||||
SHA_PATH="${TMP_DIR}/memind-portal-canary-${RELEASE_ID}.sha256"
|
||||
DRY_RUN=0
|
||||
AUTO_YES=0
|
||||
|
||||
cleanup() {
|
||||
rm -rf "${TMP_DIR}"
|
||||
}
|
||||
trap cleanup EXIT
|
||||
|
||||
usage() {
|
||||
cat <<'EOF'
|
||||
Usage:
|
||||
bash scripts/release-portal-canary-prod.sh [--dry-run] [--yes]
|
||||
|
||||
Deploys a candidate Portal beside the stable 103 runtime:
|
||||
stable Portal 127.0.0.1:8081
|
||||
canary router 127.0.0.1:18080
|
||||
candidate Portal 127.0.0.1:18081
|
||||
candidate goosed 127.0.0.1:18015
|
||||
|
||||
The stable runtime is not replaced. The 105 reverse tunnel moves to the canary
|
||||
router only after backups, candidate health, immutable identity checks, and
|
||||
automatic fallback checks pass.
|
||||
EOF
|
||||
}
|
||||
|
||||
while [[ $# -gt 0 ]]; do
|
||||
case "$1" in
|
||||
--dry-run) DRY_RUN=1 ;;
|
||||
--yes|-y) AUTO_YES=1 ;;
|
||||
-h|--help)
|
||||
usage
|
||||
exit 0
|
||||
;;
|
||||
*)
|
||||
echo "Unknown argument: $1" >&2
|
||||
usage >&2
|
||||
exit 1
|
||||
;;
|
||||
esac
|
||||
shift
|
||||
done
|
||||
|
||||
for bypass in \
|
||||
ALLOW_PORTAL_RELEASE_SCOPE_BYPASS \
|
||||
ALLOW_MINDSPACE_PUBLIC_LINK_ISSUES \
|
||||
MEMIND_RELEASE_SKIP_GATE; do
|
||||
if [[ "${!bypass:-0}" == "1" ]]; then
|
||||
echo "Production canary release forbids ${bypass}." >&2
|
||||
exit 1
|
||||
fi
|
||||
done
|
||||
|
||||
say() {
|
||||
printf '\n[%s] %s\n' "$(date +%H:%M:%S)" "$*"
|
||||
}
|
||||
|
||||
need_cmd() {
|
||||
command -v "$1" >/dev/null 2>&1 || {
|
||||
echo "Missing command: $1" >&2
|
||||
exit 1
|
||||
}
|
||||
}
|
||||
|
||||
need_cmd ssh
|
||||
need_cmd scp
|
||||
need_cmd tar
|
||||
need_cmd shasum
|
||||
|
||||
branch="$(git -C "${ROOT}" branch --show-current)"
|
||||
if [[ "${branch}" != "main" ]]; then
|
||||
echo "Canary production release requires main, got: ${branch:-detached}" >&2
|
||||
exit 1
|
||||
fi
|
||||
git -C "${ROOT}" fetch origin --prune
|
||||
if [[ "$(git -C "${ROOT}" rev-parse HEAD)" != "$(git -C "${ROOT}" rev-parse origin/main)" ]]; then
|
||||
echo "Canary production release requires HEAD to equal origin/main." >&2
|
||||
exit 1
|
||||
fi
|
||||
ALLOW_MAIN_RELEASE=1 bash "${ROOT}/scripts/check-release-ready.sh" --skip-fetch
|
||||
|
||||
say "Run release source guards"
|
||||
(
|
||||
cd "${ROOT}"
|
||||
npm test -- --test-name-pattern='publish|space|billing' >/dev/null
|
||||
npm run verify:mindspace-publish-guards >/dev/null
|
||||
npm run verify:mindspace-page-sync-guards >/dev/null
|
||||
npm run verify:h5-session-patches >/dev/null
|
||||
npm run verify:page-data >/dev/null
|
||||
npm run check:mindspace-public-links >/dev/null
|
||||
)
|
||||
|
||||
required_runtime_paths=(
|
||||
server.mjs
|
||||
memind-canary-proxy.mjs
|
||||
wechat-mp.bundle.mjs
|
||||
mindspace-sandbox-mcp.mjs
|
||||
tkmind-search-mcp.mjs
|
||||
tkmind-excel-mcp.mjs
|
||||
dist
|
||||
package.json
|
||||
scripts/run-memind-portal-candidate.sh
|
||||
scripts/run-memind-canary-proxy-prod.sh
|
||||
scripts/goosed-canary.compose.yml
|
||||
scripts/memind-portal-tunnel.sh
|
||||
)
|
||||
for relative in "${required_runtime_paths[@]}"; do
|
||||
[[ -e "${RUNTIME_ROOT}/${relative}" ]] || {
|
||||
echo "Candidate runtime is missing ${relative}; rebuild and rerun the complete Gate." >&2
|
||||
exit 1
|
||||
}
|
||||
done
|
||||
|
||||
say "Verify the complete Gate report against the exact candidate artifact"
|
||||
node "${ROOT}/scripts/verify-release-gate-report.mjs" --artifact "${RUNTIME_ROOT}"
|
||||
|
||||
say "Package the verified candidate artifact"
|
||||
tar -czf "${BUNDLE_PATH}" -C "${RUNTIME_ROOT}" .
|
||||
bundle_sha="$(shasum -a 256 "${BUNDLE_PATH}" | awk '{print $1}')"
|
||||
printf '%s %s\n' "${bundle_sha}" "$(basename "${BUNDLE_PATH}")" > "${SHA_PATH}"
|
||||
{
|
||||
echo "release_id=${RELEASE_ID}"
|
||||
echo "created_at=$(date '+%Y-%m-%d %H:%M:%S %z')"
|
||||
echo "git_head=${FULL_SHA}"
|
||||
echo "git_branch=${branch}"
|
||||
echo "artifact_tree=.runtime/portal"
|
||||
echo "artifact_bundle_sha256=${bundle_sha}"
|
||||
echo "routing=stable:8081,proxy:18080,candidate:18081,goosed-canary:18015"
|
||||
echo "canary_usernames=${CANARY_USERNAMES}"
|
||||
echo "canary_wechat_user_ids=${CANARY_WECHAT_USER_IDS}"
|
||||
echo "manual_environment_changes=none"
|
||||
} > "${MANIFEST_PATH}"
|
||||
|
||||
if [[ "${DRY_RUN}" -eq 1 ]]; then
|
||||
say "Dry-run complete; no 103 connection or production write occurred"
|
||||
ls -lh "${BUNDLE_PATH}" "${MANIFEST_PATH}" "${SHA_PATH}"
|
||||
exit 0
|
||||
fi
|
||||
|
||||
say "Run 103 read-only preflight"
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=15 "${HOST}" \
|
||||
"test -d '${STABLE_DIR}' \
|
||||
&& test -f '${STABLE_DIR}/.env' \
|
||||
&& test -f '${GOOSED_DIR}/docker-compose.prod.yml' \
|
||||
&& curl -fsS http://127.0.0.1:8081/api/status >/dev/null \
|
||||
&& /opt/homebrew/bin/docker inspect goosed-prod-1 >/dev/null \
|
||||
&& test \"\$(df -Pk '${REMOTE_ROOT}' | awk 'NR==2 {print \$4}')\" -gt 10485760"
|
||||
|
||||
if [[ "${AUTO_YES}" -ne 1 ]]; then
|
||||
say "Production canary confirmation"
|
||||
echo "Target host: ${HOST}"
|
||||
echo "Release: ${RELEASE_ID}"
|
||||
echo "Commit: ${FULL_SHA}"
|
||||
echo "Stable remains on 8081; selected immutable identities route to candidate 18081."
|
||||
read -r -p "Continue with the 103 canary release? [y/N] " confirm </dev/tty
|
||||
[[ "${confirm}" =~ ^[Yy]$ ]] || exit 0
|
||||
fi
|
||||
|
||||
say "Upload the verified candidate bundle"
|
||||
ssh -o BatchMode=yes "${HOST}" "mkdir -p '${INCOMING_DIR}' '${BACKUP_DIR}' '${CANDIDATE_BASE}'"
|
||||
scp -q "${BUNDLE_PATH}" "${MANIFEST_PATH}" "${SHA_PATH}" "${HOST}:${INCOMING_DIR}/"
|
||||
|
||||
say "Install the isolated candidate and identity router on 103"
|
||||
ssh -o BatchMode=yes "${HOST}" \
|
||||
"RELEASE_ID='${RELEASE_ID}' \
|
||||
STABLE_DIR='${STABLE_DIR}' \
|
||||
CANDIDATE_BASE='${CANDIDATE_BASE}' \
|
||||
INCOMING_DIR='${INCOMING_DIR}' \
|
||||
BACKUP_DIR='${BACKUP_DIR}' \
|
||||
GOOSED_DIR='${GOOSED_DIR}' \
|
||||
PORTAL_CANDIDATE_LABEL='${PORTAL_CANDIDATE_LABEL}' \
|
||||
CANARY_PROXY_LABEL='${CANARY_PROXY_LABEL}' \
|
||||
PORTAL_TUNNEL_LABEL='${PORTAL_TUNNEL_LABEL}' \
|
||||
MEMIND_PORTAL_TUNNEL_HOST='${MEMIND_PORTAL_TUNNEL_HOST}' \
|
||||
MEMIND_PORTAL_TUNNEL_REMOTE_PORT='${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}' \
|
||||
CANARY_USERNAMES='${CANARY_USERNAMES}' \
|
||||
CANARY_WECHAT_USER_IDS='${CANARY_WECHAT_USER_IDS}' \
|
||||
/bin/bash" <<'REMOTE_SCRIPT'
|
||||
set -euo pipefail
|
||||
|
||||
LAUNCHD_GUI="gui/$(id -u)"
|
||||
CANDIDATE_DIR="${CANDIDATE_BASE}/${RELEASE_ID}"
|
||||
BUNDLE="${INCOMING_DIR}/memind-portal-canary-${RELEASE_ID}.tar.gz"
|
||||
MANIFEST="${INCOMING_DIR}/memind-portal-canary-${RELEASE_ID}.manifest.txt"
|
||||
SHA_FILE="${INCOMING_DIR}/memind-portal-canary-${RELEASE_ID}.sha256"
|
||||
FULL_BACKUP_TAR="${BACKUP_DIR}/memind-full-${RELEASE_ID}-before-canary.tar.gz"
|
||||
PERSIST_BACKUP_TAR="${BACKUP_DIR}/memind-persisted-${RELEASE_ID}-before-canary.tar.gz"
|
||||
TUNNEL_PLIST="${HOME}/Library/LaunchAgents/${PORTAL_TUNNEL_LABEL}.plist"
|
||||
TUNNEL_PLIST_BACKUP="${BACKUP_DIR}/${PORTAL_TUNNEL_LABEL}-${RELEASE_ID}.plist"
|
||||
CANDIDATE_PLIST="${HOME}/Library/LaunchAgents/${PORTAL_CANDIDATE_LABEL}.plist"
|
||||
PROXY_PLIST="${HOME}/Library/LaunchAgents/${CANARY_PROXY_LABEL}.plist"
|
||||
SECRET_FILE="${HOME}/.config/memind/canary-router.secret"
|
||||
DOCKER_BIN="/opt/homebrew/bin/docker"
|
||||
GOOSED_COMPOSE="${GOOSED_DIR}/docker-compose.prod.yml"
|
||||
|
||||
say() {
|
||||
printf '\n[remote %s] %s\n' "$(date +%H:%M:%S)" "$*"
|
||||
}
|
||||
|
||||
candidate_healthy() {
|
||||
local headers code
|
||||
headers="$(mktemp)"
|
||||
code="$(
|
||||
curl -sS -D "${headers}" -o /dev/null -w '%{http_code}' \
|
||||
http://127.0.0.1:18081/api/status 2>/dev/null || true
|
||||
)"
|
||||
if [[ "${code}" == "200" ]] \
|
||||
&& grep -qi '^x-memind-runtime-role: candidate' "${headers}"; then
|
||||
rm -f "${headers}"
|
||||
return 0
|
||||
fi
|
||||
rm -f "${headers}"
|
||||
return 1
|
||||
}
|
||||
|
||||
write_tunnel_plist() {
|
||||
local local_port="$1"
|
||||
cat > "${TUNNEL_PLIST}" <<EOF
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key><string>${PORTAL_TUNNEL_LABEL}</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array><string>${STABLE_DIR}/scripts/memind-portal-tunnel.sh</string></array>
|
||||
<key>EnvironmentVariables</key>
|
||||
<dict>
|
||||
<key>MEMIND_PORTAL_TUNNEL_HOST</key><string>${MEMIND_PORTAL_TUNNEL_HOST}</string>
|
||||
<key>MEMIND_PORTAL_TUNNEL_LOCAL_PORT</key><string>${local_port}</string>
|
||||
<key>MEMIND_PORTAL_TUNNEL_REMOTE_PORT</key><string>${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}</string>
|
||||
</dict>
|
||||
<key>RunAtLoad</key><true/>
|
||||
<key>KeepAlive</key><true/>
|
||||
<key>StandardOutPath</key><string>${HOME}/Library/Logs/memind-portal-tunnel.log</string>
|
||||
<key>StandardErrorPath</key><string>${HOME}/Library/Logs/memind-portal-tunnel.log</string>
|
||||
</dict>
|
||||
</plist>
|
||||
EOF
|
||||
launchctl bootout "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null 2>&1 || true
|
||||
launchctl bootstrap "${LAUNCHD_GUI}" "${TUNNEL_PLIST}" >/dev/null
|
||||
launchctl enable "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null 2>&1 || true
|
||||
launchctl kickstart -k "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null
|
||||
}
|
||||
|
||||
stop_candidate_services() {
|
||||
launchctl bootout "${LAUNCHD_GUI}/${CANARY_PROXY_LABEL}" >/dev/null 2>&1 || true
|
||||
launchctl bootout "${LAUNCHD_GUI}/${PORTAL_CANDIDATE_LABEL}" >/dev/null 2>&1 || true
|
||||
"${DOCKER_BIN}" rm -f goosed-prod-canary >/dev/null 2>&1 || true
|
||||
}
|
||||
|
||||
rollback() {
|
||||
local status=$?
|
||||
trap - ERR
|
||||
say "Canary rollout failed; restoring stable-only tunnel"
|
||||
rm -f "${STABLE_DIR}/.release-drain"
|
||||
write_tunnel_plist 8081 >/dev/null 2>&1 || true
|
||||
stop_candidate_services
|
||||
exit "${status}"
|
||||
}
|
||||
trap rollback ERR
|
||||
|
||||
[[ -f "${BUNDLE}" && -f "${MANIFEST}" && -f "${SHA_FILE}" ]]
|
||||
cd "${INCOMING_DIR}"
|
||||
shasum -a 256 -c "$(basename "${SHA_FILE}")"
|
||||
|
||||
say "Create and verify the full stable backup"
|
||||
COPYFILE_DISABLE=1 tar --exclude='Memind/.tailscale/*.sock' \
|
||||
-czf "${FULL_BACKUP_TAR}" -C "$(dirname "${STABLE_DIR}")" "$(basename "${STABLE_DIR}")"
|
||||
shasum -a 256 "${FULL_BACKUP_TAR}" > "${FULL_BACKUP_TAR}.sha256"
|
||||
shasum -a 256 -c "${FULL_BACKUP_TAR}.sha256"
|
||||
gzip -t "${FULL_BACKUP_TAR}"
|
||||
tar -tzf "${FULL_BACKUP_TAR}" | grep '^Memind/.env$' >/dev/null
|
||||
|
||||
say "Create and verify the persisted-data backup"
|
||||
persist_tmp="$(mktemp -d "${TMPDIR:-/tmp}/memind-canary-persist.XXXXXX")"
|
||||
for item in .env MindSpace data users .tailscale public/plaza-covers logs; do
|
||||
if [[ -e "${STABLE_DIR}/${item}" ]]; then
|
||||
mkdir -p "${persist_tmp}/$(dirname "${item}")"
|
||||
if [[ "${item}" == ".tailscale" ]]; then
|
||||
mkdir -p "${persist_tmp}/${item}"
|
||||
(
|
||||
cd "${STABLE_DIR}/${item}"
|
||||
COPYFILE_DISABLE=1 tar --exclude='*.sock' -cf - .
|
||||
) | (
|
||||
cd "${persist_tmp}/${item}"
|
||||
COPYFILE_DISABLE=1 tar -xf -
|
||||
)
|
||||
else
|
||||
cp -a "${STABLE_DIR}/${item}" "${persist_tmp}/$(dirname "${item}")/"
|
||||
fi
|
||||
fi
|
||||
done
|
||||
COPYFILE_DISABLE=1 tar -czf "${PERSIST_BACKUP_TAR}" -C "${persist_tmp}" .
|
||||
rm -rf "${persist_tmp}"
|
||||
shasum -a 256 "${PERSIST_BACKUP_TAR}" > "${PERSIST_BACKUP_TAR}.sha256"
|
||||
shasum -a 256 -c "${PERSIST_BACKUP_TAR}.sha256"
|
||||
gzip -t "${PERSIST_BACKUP_TAR}"
|
||||
tar -tzf "${PERSIST_BACKUP_TAR}" | grep -E '^(\./)?\.env$' >/dev/null
|
||||
|
||||
if [[ -f "${TUNNEL_PLIST}" ]]; then
|
||||
cp "${TUNNEL_PLIST}" "${TUNNEL_PLIST_BACKUP}"
|
||||
fi
|
||||
|
||||
say "Extract the source-free candidate runtime"
|
||||
rm -rf "${CANDIDATE_DIR}"
|
||||
mkdir -p "${CANDIDATE_DIR}"
|
||||
tar -xzf "${BUNDLE}" -C "${CANDIDATE_DIR}"
|
||||
cp "${MANIFEST}" "${CANDIDATE_DIR}/.release-manifest.txt"
|
||||
chmod 755 \
|
||||
"${CANDIDATE_DIR}/scripts/run-memind-portal-candidate.sh" \
|
||||
"${CANDIDATE_DIR}/scripts/run-memind-canary-proxy-prod.sh"
|
||||
|
||||
say "Stop an older canary without touching stable Portal 8081"
|
||||
if curl -fsS http://127.0.0.1:18080/api/status >/dev/null 2>&1; then
|
||||
write_tunnel_plist 8081
|
||||
sleep 2
|
||||
fi
|
||||
stop_candidate_services
|
||||
|
||||
say "Start an isolated goosed candidate on 18015"
|
||||
(
|
||||
cd "${GOOSED_DIR}"
|
||||
set -a
|
||||
# shellcheck disable=SC1091
|
||||
source .env
|
||||
set +a
|
||||
export CANDIDATE_RUNTIME_DIR="${CANDIDATE_DIR}"
|
||||
"${DOCKER_BIN}" compose -p goosed-prod \
|
||||
-f "${GOOSED_COMPOSE}" \
|
||||
-f "${CANDIDATE_DIR}/scripts/goosed-canary.compose.yml" \
|
||||
up -d --no-deps goosed-canary
|
||||
)
|
||||
for _ in $(seq 1 60); do
|
||||
if [[ "$(curl -skS -m 5 https://127.0.0.1:18015/status 2>/dev/null || true)" == "ok" ]]; then
|
||||
break
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
[[ "$(curl -skS -m 5 https://127.0.0.1:18015/status 2>/dev/null || true)" == "ok" ]]
|
||||
"${DOCKER_BIN}" exec goosed-prod-canary \
|
||||
sh -lc 'test -x /usr/local/bin/node && test -f /opt/portal/mindspace-sandbox-mcp.mjs'
|
||||
|
||||
say "Start the passive candidate Portal on 18081"
|
||||
cat > "${CANDIDATE_PLIST}" <<EOF
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key><string>${PORTAL_CANDIDATE_LABEL}</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array><string>${CANDIDATE_DIR}/scripts/run-memind-portal-candidate.sh</string></array>
|
||||
<key>EnvironmentVariables</key>
|
||||
<dict>
|
||||
<key>MEMIND_CANARY_STABLE_ROOT</key><string>${STABLE_DIR}</string>
|
||||
<key>MEMIND_CANARY_RELEASE_ID</key><string>${RELEASE_ID}</string>
|
||||
<key>MEMIND_CANARY_CANDIDATE_PORT</key><string>18081</string>
|
||||
<key>MEMIND_CANARY_GOOSED_URL</key><string>https://127.0.0.1:18015</string>
|
||||
</dict>
|
||||
<key>RunAtLoad</key><true/>
|
||||
<key>KeepAlive</key><true/>
|
||||
<key>ThrottleInterval</key><integer>5</integer>
|
||||
<key>StandardOutPath</key><string>${HOME}/Library/Logs/memind-portal-candidate.log</string>
|
||||
<key>StandardErrorPath</key><string>${HOME}/Library/Logs/memind-portal-candidate.log</string>
|
||||
</dict>
|
||||
</plist>
|
||||
EOF
|
||||
launchctl bootstrap "${LAUNCHD_GUI}" "${CANDIDATE_PLIST}" >/dev/null
|
||||
launchctl kickstart -k "${LAUNCHD_GUI}/${PORTAL_CANDIDATE_LABEL}" >/dev/null
|
||||
for _ in $(seq 1 60); do
|
||||
candidate_healthy && break
|
||||
sleep 2
|
||||
done
|
||||
candidate_healthy
|
||||
|
||||
say "Validate immutable canary selectors against the production identity database"
|
||||
mkdir -p "$(dirname "${SECRET_FILE}")"
|
||||
if [[ ! -s "${SECRET_FILE}" ]]; then
|
||||
umask 077
|
||||
/usr/bin/openssl rand -hex 32 > "${SECRET_FILE}"
|
||||
fi
|
||||
chmod 600 "${SECRET_FILE}"
|
||||
(
|
||||
set -a
|
||||
# shellcheck disable=SC1091
|
||||
source "${STABLE_DIR}/.env"
|
||||
set +a
|
||||
export MEMIND_PORTAL_H5_ROOT="${STABLE_DIR}"
|
||||
export MEMIND_RELEASE_CANARY_USERNAMES="${CANARY_USERNAMES}"
|
||||
export MEMIND_RELEASE_CANARY_WECHAT_USER_IDS="${CANARY_WECHAT_USER_IDS}"
|
||||
/opt/homebrew/opt/node@24/bin/node "${CANDIDATE_DIR}/memind-canary-proxy.mjs" --check
|
||||
)
|
||||
|
||||
say "Start the fail-closed identity router on 18080"
|
||||
cat > "${PROXY_PLIST}" <<EOF
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key><string>${CANARY_PROXY_LABEL}</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array><string>${CANDIDATE_DIR}/scripts/run-memind-canary-proxy-prod.sh</string></array>
|
||||
<key>EnvironmentVariables</key>
|
||||
<dict>
|
||||
<key>MEMIND_CANARY_STABLE_ROOT</key><string>${STABLE_DIR}</string>
|
||||
<key>MEMIND_CANARY_SECRET_FILE</key><string>${SECRET_FILE}</string>
|
||||
<key>MEMIND_RELEASE_CANARY_USERNAMES</key><string>${CANARY_USERNAMES}</string>
|
||||
<key>MEMIND_RELEASE_CANARY_WECHAT_USER_IDS</key><string>${CANARY_WECHAT_USER_IDS}</string>
|
||||
</dict>
|
||||
<key>RunAtLoad</key><true/>
|
||||
<key>KeepAlive</key><true/>
|
||||
<key>ThrottleInterval</key><integer>5</integer>
|
||||
<key>StandardOutPath</key><string>${HOME}/Library/Logs/memind-canary-proxy.log</string>
|
||||
<key>StandardErrorPath</key><string>${HOME}/Library/Logs/memind-canary-proxy.log</string>
|
||||
</dict>
|
||||
</plist>
|
||||
EOF
|
||||
launchctl bootstrap "${LAUNCHD_GUI}" "${PROXY_PLIST}" >/dev/null
|
||||
launchctl kickstart -k "${LAUNCHD_GUI}/${CANARY_PROXY_LABEL}" >/dev/null
|
||||
sleep 2
|
||||
diagnostic_secret="$(tr -d '\r\n' < "${SECRET_FILE}")"
|
||||
diagnostic="$(
|
||||
curl -fsS \
|
||||
-H "X-Memind-Canary-Secret: ${diagnostic_secret}" \
|
||||
http://127.0.0.1:18080/__memind_canary/health
|
||||
)"
|
||||
printf '%s' "${diagnostic}" | /opt/homebrew/opt/node@24/bin/node --input-type=module -e '
|
||||
let input = "";
|
||||
process.stdin.on("data", (chunk) => { input += chunk; });
|
||||
process.stdin.on("end", () => {
|
||||
const status = JSON.parse(input);
|
||||
if (!status.ok || !status.stable || !status.candidate) process.exit(1);
|
||||
if (status.policy.usernames < 1 || status.policy.wechatUserIds < 1) process.exit(1);
|
||||
});
|
||||
'
|
||||
for probe_query in 'username=john' 'wechat_user_id=wx_ul610et8'; do
|
||||
route_probe="$(
|
||||
curl -fsS \
|
||||
-H "X-Memind-Canary-Secret: ${diagnostic_secret}" \
|
||||
"http://127.0.0.1:18080/__memind_canary/route-probe?${probe_query}"
|
||||
)"
|
||||
printf '%s' "${route_probe}" | /opt/homebrew/opt/node@24/bin/node --input-type=module -e '
|
||||
let input = "";
|
||||
process.stdin.on("data", (chunk) => { input += chunk; });
|
||||
process.stdin.on("end", () => {
|
||||
const status = JSON.parse(input);
|
||||
if (status.configuredTarget !== "candidate") process.exit(1);
|
||||
if (status.candidate !== true || status.route !== "candidate") process.exit(1);
|
||||
});
|
||||
'
|
||||
done
|
||||
|
||||
say "Drain active Agent runs for the bounded tunnel switch"
|
||||
touch "${STABLE_DIR}/.release-drain"
|
||||
deadline=$(( $(date +%s) + 120 ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
queue="$(
|
||||
/opt/homebrew/opt/node@24/bin/node \
|
||||
"${STABLE_DIR}/scripts/agent-run-worker.mjs" --status 2>/dev/null || true
|
||||
)"
|
||||
counts="$(
|
||||
printf '%s' "${queue}" | /opt/homebrew/opt/node@24/bin/node --input-type=module -e '
|
||||
let input = "";
|
||||
process.stdin.on("data", (chunk) => { input += chunk; });
|
||||
process.stdin.on("end", () => {
|
||||
try {
|
||||
const queue = JSON.parse(input).queue || {};
|
||||
console.log(`${Number(queue.inFlight || 0)} ${Number(queue.pendingDispatches || 0)}`);
|
||||
} catch {
|
||||
console.log("unknown unknown");
|
||||
}
|
||||
});
|
||||
'
|
||||
)"
|
||||
if [[ "${counts}" == "0 0" ]]; then
|
||||
break
|
||||
fi
|
||||
sleep 2
|
||||
done
|
||||
[[ "${counts:-unknown unknown}" == "0 0" ]]
|
||||
|
||||
say "Switch only the reverse tunnel from stable 8081 to identity router 18080"
|
||||
write_tunnel_plist 18080
|
||||
sleep 2
|
||||
remote_code="$(
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=10 "${MEMIND_PORTAL_TUNNEL_HOST}" \
|
||||
"curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}/api/status" \
|
||||
2>/dev/null || true
|
||||
)"
|
||||
[[ "${remote_code}" == "200" ]]
|
||||
rm -f "${STABLE_DIR}/.release-drain"
|
||||
|
||||
say "Verify candidate failure automatically falls back to stable"
|
||||
launchctl bootout "${LAUNCHD_GUI}/${PORTAL_CANDIDATE_LABEL}" >/dev/null
|
||||
for _ in $(seq 1 20); do
|
||||
if ! curl -fsS http://127.0.0.1:18081/api/status >/dev/null 2>&1; then
|
||||
break
|
||||
fi
|
||||
sleep 1
|
||||
done
|
||||
fallback_probe="$(
|
||||
curl -fsS \
|
||||
-H "X-Memind-Canary-Secret: ${diagnostic_secret}" \
|
||||
'http://127.0.0.1:18080/__memind_canary/route-probe?username=john'
|
||||
)"
|
||||
printf '%s' "${fallback_probe}" | /opt/homebrew/opt/node@24/bin/node --input-type=module -e '
|
||||
let input = "";
|
||||
process.stdin.on("data", (chunk) => { input += chunk; });
|
||||
process.stdin.on("end", () => {
|
||||
const status = JSON.parse(input);
|
||||
if (status.configuredTarget !== "candidate") process.exit(1);
|
||||
if (status.candidate !== false || status.route !== "stable") process.exit(1);
|
||||
});
|
||||
'
|
||||
launchctl bootstrap "${LAUNCHD_GUI}" "${CANDIDATE_PLIST}" >/dev/null
|
||||
launchctl kickstart -k "${LAUNCHD_GUI}/${PORTAL_CANDIDATE_LABEL}" >/dev/null
|
||||
for _ in $(seq 1 60); do
|
||||
candidate_healthy && break
|
||||
sleep 2
|
||||
done
|
||||
candidate_healthy
|
||||
final_diagnostic="$(
|
||||
curl -fsS \
|
||||
-H "X-Memind-Canary-Secret: ${diagnostic_secret}" \
|
||||
http://127.0.0.1:18080/__memind_canary/health
|
||||
)"
|
||||
printf '%s' "${final_diagnostic}" | /opt/homebrew/opt/node@24/bin/node --input-type=module -e '
|
||||
let input = "";
|
||||
process.stdin.on("data", (chunk) => { input += chunk; });
|
||||
process.stdin.on("end", () => {
|
||||
const status = JSON.parse(input);
|
||||
if (!status.ok || !status.stable || !status.candidate) process.exit(1);
|
||||
});
|
||||
'
|
||||
|
||||
trap - ERR
|
||||
say "103 production canary is active"
|
||||
printf 'release_id=%s\n' "${RELEASE_ID}"
|
||||
printf 'candidate_dir=%s\n' "${CANDIDATE_DIR}"
|
||||
printf 'full_backup=%s\n' "${FULL_BACKUP_TAR}"
|
||||
printf 'persist_backup=%s\n' "${PERSIST_BACKUP_TAR}"
|
||||
printf 'stable_health=http://127.0.0.1:8081/api/status\n'
|
||||
printf 'proxy_health=http://127.0.0.1:18080/__memind_canary/health\n'
|
||||
printf 'candidate_health=http://127.0.0.1:18081/api/status\n'
|
||||
printf 'candidate_goosed=https://127.0.0.1:18015/status\n'
|
||||
REMOTE_SCRIPT
|
||||
|
||||
say "103 canary release completed"
|
||||
printf 'release_id=%s\n' "${RELEASE_ID}"
|
||||
printf 'git_head=%s\n' "${FULL_SHA}"
|
||||
@@ -84,8 +84,9 @@ if [[ "${ALLOW_MINDSPACE_PUBLIC_LINK_ISSUES:-0}" == "1" ]]; then
|
||||
fi
|
||||
|
||||
if [[ "${DRY_RUN}" -eq 0 ]]; then
|
||||
echo "当前脚本执行整包替换，不是用户级灰度入口；在候选 runtime + 灰度代理上线前禁止生产发布。" >&2
|
||||
echo "请先使用 docs/release-canary-103.md 规定的灰度流程，并由守门员重新授权。" >&2
|
||||
echo "当前脚本执行 8081 整包晋升，不是用户级灰度入口。" >&2
|
||||
echo "在同一候选完成 103 灰度验收且晋升证据校验落地前，禁止非 dry-run。" >&2
|
||||
echo "首次生产动作请使用 scripts/release-portal-canary-prod.sh。" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
@@ -165,7 +166,7 @@ fi
|
||||
|
||||
verify_runtime_artifact() {
|
||||
local missing=0
|
||||
for required in server.mjs wechat-mp.bundle.mjs mindspace-sandbox-mcp.mjs tkmind-search-mcp.mjs tkmind-excel-mcp.mjs mindspace-public-links.mjs dist package.json scripts/run-memind-portal-prod.sh scripts/check-mindspace-public-links.mjs scripts/load-env.mjs scripts/wechat-mp-menu.mjs scripts/memind-portal-tunnel.sh; do
|
||||
for required in server.mjs memind-canary-proxy.mjs wechat-mp.bundle.mjs mindspace-sandbox-mcp.mjs tkmind-search-mcp.mjs tkmind-excel-mcp.mjs mindspace-public-links.mjs dist package.json scripts/run-memind-portal-prod.sh scripts/run-memind-portal-candidate.sh scripts/run-memind-canary-proxy-prod.sh scripts/goosed-canary.compose.yml scripts/check-mindspace-public-links.mjs scripts/load-env.mjs scripts/wechat-mp-menu.mjs scripts/memind-portal-tunnel.sh; do
|
||||
if [[ ! -e "${RUNTIME_ROOT}/${required}" ]]; then
|
||||
echo "runtime 产物缺失: ${RUNTIME_ROOT}/${required}" >&2
|
||||
missing=1
|
||||
|
||||
Executable
+81
@@ -0,0 +1,81 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
HOST="${STUDIO_HOST:-58.38.22.103}"
|
||||
STABLE_DIR="${STUDIO_REMOTE_ROOT:-/Users/john/Project}/Memind"
|
||||
PORTAL_CANDIDATE_LABEL="cn.tkmind.memind-portal-candidate"
|
||||
CANARY_PROXY_LABEL="cn.tkmind.memind-canary-proxy"
|
||||
PORTAL_TUNNEL_LABEL="cn.tkmind.memind-portal-tunnel"
|
||||
MEMIND_PORTAL_TUNNEL_HOST="${MEMIND_PORTAL_TUNNEL_HOST:-ssh105-public}"
|
||||
MEMIND_PORTAL_TUNNEL_REMOTE_PORT="${MEMIND_PORTAL_TUNNEL_REMOTE_PORT:-19081}"
|
||||
AUTO_YES=0
|
||||
|
||||
if [[ "${1:-}" == "--yes" || "${1:-}" == "-y" ]]; then
|
||||
AUTO_YES=1
|
||||
elif [[ $# -gt 0 ]]; then
|
||||
echo "Usage: bash scripts/rollback-portal-canary-prod.sh [--yes]" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
if [[ "${AUTO_YES}" -ne 1 ]]; then
|
||||
echo "This restores the 105 tunnel to stable Portal 8081 and stops only canary services."
|
||||
read -r -p "Continue with 103 canary rollback? [y/N] " confirm </dev/tty
|
||||
[[ "${confirm}" =~ ^[Yy]$ ]] || exit 0
|
||||
fi
|
||||
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=15 "${HOST}" \
|
||||
"STABLE_DIR='${STABLE_DIR}' \
|
||||
PORTAL_CANDIDATE_LABEL='${PORTAL_CANDIDATE_LABEL}' \
|
||||
CANARY_PROXY_LABEL='${CANARY_PROXY_LABEL}' \
|
||||
PORTAL_TUNNEL_LABEL='${PORTAL_TUNNEL_LABEL}' \
|
||||
MEMIND_PORTAL_TUNNEL_HOST='${MEMIND_PORTAL_TUNNEL_HOST}' \
|
||||
MEMIND_PORTAL_TUNNEL_REMOTE_PORT='${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}' \
|
||||
/bin/bash" <<'REMOTE'
|
||||
set -euo pipefail
|
||||
|
||||
LAUNCHD_GUI="gui/$(id -u)"
|
||||
TUNNEL_PLIST="${HOME}/Library/LaunchAgents/${PORTAL_TUNNEL_LABEL}.plist"
|
||||
|
||||
cat > "${TUNNEL_PLIST}" <<EOF
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key><string>${PORTAL_TUNNEL_LABEL}</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array><string>${STABLE_DIR}/scripts/memind-portal-tunnel.sh</string></array>
|
||||
<key>EnvironmentVariables</key>
|
||||
<dict>
|
||||
<key>MEMIND_PORTAL_TUNNEL_HOST</key><string>${MEMIND_PORTAL_TUNNEL_HOST}</string>
|
||||
<key>MEMIND_PORTAL_TUNNEL_LOCAL_PORT</key><string>8081</string>
|
||||
<key>MEMIND_PORTAL_TUNNEL_REMOTE_PORT</key><string>${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}</string>
|
||||
</dict>
|
||||
<key>RunAtLoad</key><true/>
|
||||
<key>KeepAlive</key><true/>
|
||||
<key>StandardOutPath</key><string>${HOME}/Library/Logs/memind-portal-tunnel.log</string>
|
||||
<key>StandardErrorPath</key><string>${HOME}/Library/Logs/memind-portal-tunnel.log</string>
|
||||
</dict>
|
||||
</plist>
|
||||
EOF
|
||||
|
||||
launchctl bootout "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null 2>&1 || true
|
||||
launchctl bootstrap "${LAUNCHD_GUI}" "${TUNNEL_PLIST}" >/dev/null
|
||||
launchctl kickstart -k "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null
|
||||
launchctl bootout "${LAUNCHD_GUI}/${CANARY_PROXY_LABEL}" >/dev/null 2>&1 || true
|
||||
launchctl bootout "${LAUNCHD_GUI}/${PORTAL_CANDIDATE_LABEL}" >/dev/null 2>&1 || true
|
||||
/opt/homebrew/bin/docker rm -f goosed-prod-canary >/dev/null 2>&1 || true
|
||||
rm -f "${STABLE_DIR}/.release-drain"
|
||||
|
||||
curl -fsS http://127.0.0.1:8081/api/status >/dev/null
|
||||
sleep 2
|
||||
remote_code="$(
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=10 "${MEMIND_PORTAL_TUNNEL_HOST}" \
|
||||
"curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}/api/status" \
|
||||
2>/dev/null || true
|
||||
)"
|
||||
[[ "${remote_code}" == "200" ]]
|
||||
printf 'stable_port=8081\n'
|
||||
printf 'remote_tunnel_status=%s\n' "${remote_code}"
|
||||
REMOTE
|
||||
|
||||
echo "103 canary rollback completed; stable Portal remains active on 8081."
|
||||
Executable
+35
@@ -0,0 +1,35 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
STABLE_ROOT="${MEMIND_CANARY_STABLE_ROOT:-/Users/john/Project/Memind}"
|
||||
SECRET_FILE="${MEMIND_CANARY_SECRET_FILE:-${HOME}/.config/memind/canary-router.secret}"
|
||||
|
||||
if [[ ! -d "${STABLE_ROOT}" || ! -f "${STABLE_ROOT}/.env" ]]; then
|
||||
echo "[canary-proxy] stable persistent root is unavailable: ${STABLE_ROOT}" >&2
|
||||
exit 1
|
||||
fi
|
||||
if [[ ! -r "${SECRET_FILE}" ]]; then
|
||||
echo "[canary-proxy] diagnostic secret file is unavailable: ${SECRET_FILE}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
set -a
|
||||
# shellcheck disable=SC1091
|
||||
source "${STABLE_ROOT}/.env"
|
||||
set +a
|
||||
|
||||
export NODE_ENV=production
|
||||
export MEMIND_PORTAL_H5_ROOT="${STABLE_ROOT}"
|
||||
export MEMIND_CANARY_DIAGNOSTIC_SECRET="$(tr -d '\r\n' < "${SECRET_FILE}")"
|
||||
export MEMIND_CANARY_PROXY_HOST=127.0.0.1
|
||||
export MEMIND_CANARY_PROXY_PORT="${MEMIND_CANARY_PROXY_PORT:-18080}"
|
||||
export MEMIND_CANARY_STABLE_URL="${MEMIND_CANARY_STABLE_URL:-http://127.0.0.1:8081}"
|
||||
export MEMIND_CANARY_CANDIDATE_URL="${MEMIND_CANARY_CANDIDATE_URL:-http://127.0.0.1:18081}"
|
||||
|
||||
NODE_BIN="${NODE_BIN:-/opt/homebrew/opt/node@24/bin/node}"
|
||||
if [[ ! -x "${NODE_BIN}" ]]; then
|
||||
NODE_BIN="$(command -v node)"
|
||||
fi
|
||||
|
||||
exec "${NODE_BIN}" "${ROOT}/memind-canary-proxy.mjs"
|
||||
Executable
+43
@@ -0,0 +1,43 @@
|
||||
#!/usr/bin/env bash
|
||||
set -euo pipefail
|
||||
|
||||
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
|
||||
STABLE_ROOT="${MEMIND_CANARY_STABLE_ROOT:-/Users/john/Project/Memind}"
|
||||
|
||||
if [[ ! -d "${STABLE_ROOT}" || ! -f "${STABLE_ROOT}/.env" ]]; then
|
||||
echo "[portal-candidate] stable persistent root is unavailable: ${STABLE_ROOT}" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
set -a
|
||||
# shellcheck disable=SC1091
|
||||
source "${STABLE_ROOT}/.env"
|
||||
set +a
|
||||
|
||||
export NODE_ENV=production
|
||||
export MEMIND_PORTAL_H5_ROOT="${STABLE_ROOT}"
|
||||
export MEMIND_RUNTIME_ROLE=candidate
|
||||
export MEMIND_CANARY_PASSIVE_RUNTIME=1
|
||||
export MEMIND_RUNTIME_BUILD_ID="${MEMIND_CANARY_RELEASE_ID:?MEMIND_CANARY_RELEASE_ID is required}"
|
||||
export MEMIND_AGENT_RUN_WORKER_ID="portal-candidate-${MEMIND_RUNTIME_BUILD_ID}"
|
||||
export MEMIND_WORKSPACE_MAINTENANCE=0
|
||||
export H5_REMINDER_WORKER_ENABLED=0
|
||||
export H5_PORT="${MEMIND_CANARY_CANDIDATE_PORT:-18081}"
|
||||
export H5_HOST=127.0.0.1
|
||||
export H5_PUBLIC_BASE_URL="${H5_PUBLIC_BASE_URL:-https://m.tkmind.cn}"
|
||||
export TKMIND_API_TARGETS="${MEMIND_CANARY_GOOSED_URL:-https://127.0.0.1:18015}"
|
||||
export TKMIND_API_TARGET="${MEMIND_CANARY_GOOSED_URL:-https://127.0.0.1:18015}"
|
||||
unset TKMIND_API_TARGET_1
|
||||
|
||||
NODE_BIN="${NODE_BIN:-/opt/homebrew/opt/node@24/bin/node}"
|
||||
if [[ ! -x "${NODE_BIN}" ]]; then
|
||||
NODE_BIN="$(command -v node)"
|
||||
fi
|
||||
|
||||
port="${H5_PORT}"
|
||||
if lsof -tiTCP:"${port}" -sTCP:LISTEN >/dev/null 2>&1; then
|
||||
echo "[portal-candidate] port ${port} is already in use; refusing to replace it" >&2
|
||||
exit 1
|
||||
fi
|
||||
|
||||
exec "${NODE_BIN}" "${ROOT}/server.mjs"
|
||||
@@ -8,6 +8,7 @@ import { createMindSpaceAuditWriter } from '../mindspace-audit.mjs';
|
||||
import { createMindSpaceImageGenerationService } from '../mindspace-image-generation.mjs';
|
||||
import { createMindSpaceImageReviewService } from '../mindspace-image-review.mjs';
|
||||
import { createWordFilterService } from '../word-filter.mjs';
|
||||
import { isPassiveCanaryRuntime } from './portal-runtime-role.mjs';
|
||||
|
||||
async function createPgExperienceServiceFromModule(options) {
|
||||
const { createPgExperienceService } = await import(
|
||||
@@ -108,28 +109,32 @@ export async function bootstrapPortalAgentServices({
|
||||
const mindSpaceAudit =
|
||||
createMindSpaceAuditWriterFn(pool);
|
||||
|
||||
mindSpaceRuntimeAdapter.startBackgroundJobs({
|
||||
publicationCleanupIntervalMs: 60 * 1000,
|
||||
agentWorker: runtime.agentWorker,
|
||||
agentRunner: mindSpaceAgentRunner,
|
||||
workspaceMaintenanceEnabled,
|
||||
publishRoot: runtime.publishRoot,
|
||||
startWorkspaceThumbnailWatcher,
|
||||
startWorkspaceAssetSyncWatcher,
|
||||
syncUserWorkspaceByDirKey: async (
|
||||
dirKey,
|
||||
options,
|
||||
) => {
|
||||
const userId =
|
||||
await resolveUserIdByDirKey(dirKey);
|
||||
if (!userId) return;
|
||||
await mindSpaceAssets.syncWorkspaceAssets(
|
||||
userId,
|
||||
if (isPassiveCanaryRuntime(env)) {
|
||||
logger.log('Passive candidate runtime: MindSpace background jobs disabled');
|
||||
} else {
|
||||
mindSpaceRuntimeAdapter.startBackgroundJobs({
|
||||
publicationCleanupIntervalMs: 60 * 1000,
|
||||
agentWorker: runtime.agentWorker,
|
||||
agentRunner: mindSpaceAgentRunner,
|
||||
workspaceMaintenanceEnabled,
|
||||
publishRoot: runtime.publishRoot,
|
||||
startWorkspaceThumbnailWatcher,
|
||||
startWorkspaceAssetSyncWatcher,
|
||||
syncUserWorkspaceByDirKey: async (
|
||||
dirKey,
|
||||
options,
|
||||
);
|
||||
},
|
||||
expireStaleUploadsIntervalMs: 5 * 60 * 1000,
|
||||
});
|
||||
) => {
|
||||
const userId =
|
||||
await resolveUserIdByDirKey(dirKey);
|
||||
if (!userId) return;
|
||||
await mindSpaceAssets.syncWorkspaceAssets(
|
||||
userId,
|
||||
options,
|
||||
);
|
||||
},
|
||||
expireStaleUploadsIntervalMs: 5 * 60 * 1000,
|
||||
});
|
||||
}
|
||||
|
||||
await userAuth.ensureAdminUser();
|
||||
const userDataSpaceBackfill =
|
||||
|
||||
@@ -424,3 +424,25 @@ test('can disable relay bootstrap for backend-LLM-only isolated runs', async ()
|
||||
assert.equal(setup.calls.some(([name]) => name === 'relay-bootstrap'), false);
|
||||
assert.equal(setup.calls.some(([name]) => name === 'provider-sync'), true);
|
||||
});
|
||||
|
||||
test('passive candidate runtime does not start MindSpace background jobs', async () => {
|
||||
const setup = createSetup({
|
||||
env: {
|
||||
EXPERIENCE_PG_URL: 'postgres://experience',
|
||||
MEMIND_RUNTIME_ROLE: 'candidate',
|
||||
MEMIND_CANARY_PASSIVE_RUNTIME: '1',
|
||||
},
|
||||
});
|
||||
|
||||
const result = await bootstrapPortalAgentServices(setup.options);
|
||||
await Promise.all([result.relayBootstrapTask, result.providerSyncTask]);
|
||||
|
||||
assert.equal(setup.calls.some(([name]) => name === 'start-background'), false);
|
||||
assert.ok(
|
||||
setup.calls.some(
|
||||
([name, message]) =>
|
||||
name === 'log'
|
||||
&& message === 'Passive candidate runtime: MindSpace background jobs disabled',
|
||||
),
|
||||
);
|
||||
});
|
||||
|
||||
@@ -23,6 +23,7 @@ import { createPlazaOpsService } from '../plaza-ops.mjs';
|
||||
import { createPlazaPostService, formatPostRow } from '../plaza-posts.mjs';
|
||||
import { createPlazaRecommendService } from '../plaza-recommend.mjs';
|
||||
import { createPlazaRedis } from '../plaza-redis.mjs';
|
||||
import { isPassiveCanaryRuntime } from './portal-runtime-role.mjs';
|
||||
import { createPlazaSeoService } from '../plaza-seo.mjs';
|
||||
import {
|
||||
startPlazaTasks,
|
||||
@@ -287,12 +288,16 @@ export async function bootstrapPortalDomainServices({
|
||||
invalidateFeedCaches: () =>
|
||||
plazaRedis?.invalidateFeedCaches?.(),
|
||||
});
|
||||
startPlazaTasksFn({
|
||||
pool,
|
||||
plazaRedis,
|
||||
recalculateHotScores: recalculateHotScoresFn,
|
||||
writebackPublications: writebackPublicationsFn,
|
||||
});
|
||||
if (isPassiveCanaryRuntime(env)) {
|
||||
logger.log('Passive candidate runtime: Plaza background tasks disabled');
|
||||
} else {
|
||||
startPlazaTasksFn({
|
||||
pool,
|
||||
plazaRedis,
|
||||
recalculateHotScores: recalculateHotScoresFn,
|
||||
writebackPublications: writebackPublicationsFn,
|
||||
});
|
||||
}
|
||||
|
||||
return {
|
||||
mindSearchConfigService,
|
||||
|
||||
@@ -416,3 +416,24 @@ test('domain bootstrap preserves directory and Plaza cross-service callbacks', a
|
||||
],
|
||||
);
|
||||
});
|
||||
|
||||
test('passive candidate runtime does not start Plaza background tasks', async () => {
|
||||
const setup = createBootstrapSetup({
|
||||
env: {
|
||||
H5_DEFAULT_TIMEZONE: 'Asia/Singapore',
|
||||
MEMIND_RUNTIME_ROLE: 'candidate',
|
||||
MEMIND_CANARY_PASSIVE_RUNTIME: '1',
|
||||
},
|
||||
});
|
||||
|
||||
await bootstrapPortalDomainServices(setup.dependencies);
|
||||
|
||||
assert.equal(setup.calls.some((call) => call.kind === 'plaza-tasks'), false);
|
||||
assert.ok(
|
||||
setup.calls.some(
|
||||
(call) =>
|
||||
call.kind === 'log'
|
||||
&& call.message === 'Passive candidate runtime: Plaza background tasks disabled',
|
||||
),
|
||||
);
|
||||
});
|
||||
|
||||
@@ -18,6 +18,7 @@ import { createOrchestratorAdminConfigService } from '../services/orchestrator/a
|
||||
import { createWorkflowShadowObserver } from '../services/orchestrator/shadow-observer.mjs';
|
||||
import { createTkmindProxy } from '../tkmind-proxy.mjs';
|
||||
import { createToolGateway } from '../tool-gateway.mjs';
|
||||
import { isPassiveCanaryRuntime } from './portal-runtime-role.mjs';
|
||||
|
||||
function isEnabledFlag(value, fallback = '') {
|
||||
return ['1', 'true', 'yes', 'on'].includes(
|
||||
@@ -433,11 +434,12 @@ export function bootstrapPortalGatewayServices({
|
||||
),
|
||||
workerIdentity,
|
||||
});
|
||||
const agentRunRecoveryTimer =
|
||||
startAgentRunRecoveryLoopFn(
|
||||
agentRunGateway,
|
||||
{ env, logger: console },
|
||||
);
|
||||
const agentRunRecoveryTimer = isPassiveCanaryRuntime(env)
|
||||
? null
|
||||
: startAgentRunRecoveryLoopFn(
|
||||
agentRunGateway,
|
||||
{ env, logger: console },
|
||||
);
|
||||
|
||||
return {
|
||||
tkmindProxy,
|
||||
|
||||
@@ -288,6 +288,25 @@ test('preserves memory, page sync, and busy callbacks', async () => {
|
||||
);
|
||||
});
|
||||
|
||||
test('passive candidate runtime disables the singleton Agent recovery loop', () => {
|
||||
let recoveryStarts = 0;
|
||||
const setup = createSetup({
|
||||
env: {
|
||||
MEMIND_RUNTIME_ROLE: 'candidate',
|
||||
MEMIND_CANARY_PASSIVE_RUNTIME: '1',
|
||||
},
|
||||
startAgentRunRecoveryLoopFn() {
|
||||
recoveryStarts += 1;
|
||||
return { id: 'recovery' };
|
||||
},
|
||||
});
|
||||
|
||||
const result = bootstrapPortalGatewayServices(setup.options);
|
||||
|
||||
assert.equal(recoveryStarts, 0);
|
||||
assert.equal(result.agentRunRecoveryTimer, null);
|
||||
});
|
||||
|
||||
test('keeps memory observation optional and parses disabled dispatch', async () => {
|
||||
const setup = createSetup({
|
||||
env: {
|
||||
|
||||
@@ -11,6 +11,7 @@ import { createPageEditSessionService } from '../mindspace-page-edit-session.mjs
|
||||
import { resolveMindSpaceRuntimeConfig } from '../mindspace-runtime-config.mjs';
|
||||
import { createNotificationDispatcher } from '../notification-dispatcher.mjs';
|
||||
import { startScheduleReminderWorker } from '../schedule-reminder-worker.mjs';
|
||||
import { isPassiveCanaryRuntime } from './portal-runtime-role.mjs';
|
||||
import { loadWechatMpModule } from '../wechat-mp-loader.mjs';
|
||||
|
||||
export async function bootstrapPortalIntegrationServices({
|
||||
@@ -224,6 +225,7 @@ export async function bootstrapPortalIntegrationServices({
|
||||
|
||||
let scheduleReminderWorker = null;
|
||||
if (
|
||||
!isPassiveCanaryRuntime(env) &&
|
||||
env.H5_REMINDER_WORKER_ENABLED === '1' &&
|
||||
wechatMpService?.enabled &&
|
||||
scheduleService
|
||||
@@ -237,7 +239,7 @@ export async function bootstrapPortalIntegrationServices({
|
||||
}
|
||||
|
||||
let subscriptionExpiryTimer = null;
|
||||
if (subscriptionService) {
|
||||
if (subscriptionService && !isPassiveCanaryRuntime(env)) {
|
||||
subscriptionExpiryTimer = setIntervalFn(
|
||||
async () => {
|
||||
try {
|
||||
|
||||
@@ -507,3 +507,24 @@ test('keeps optional integrations disabled and contains timer failures', async (
|
||||
),
|
||||
);
|
||||
});
|
||||
|
||||
test('passive candidate runtime disables singleton reminder and subscription timers', async () => {
|
||||
const setup = createSetup({
|
||||
env: {
|
||||
H5_SCHEDULE_ENABLED: '1',
|
||||
H5_REMINDER_WORKER_ENABLED: '1',
|
||||
MEMIND_RUNTIME_ROLE: 'candidate',
|
||||
MEMIND_CANARY_PASSIVE_RUNTIME: '1',
|
||||
},
|
||||
});
|
||||
|
||||
const result = await bootstrapPortalIntegrationServices(setup.options);
|
||||
const captured = setup.getCaptured();
|
||||
|
||||
assert.equal(result.scheduleReminderWorker, null);
|
||||
assert.equal(result.subscriptionExpiryTimer, null);
|
||||
assert.equal(captured.reminderOptions, undefined);
|
||||
assert.equal(captured.timerCallback, undefined);
|
||||
assert.equal(setup.calls.some(([name]) => name === 'reminder-worker'), false);
|
||||
assert.equal(setup.calls.some(([name]) => name === 'set-interval'), false);
|
||||
});
|
||||
|
||||
@@ -0,0 +1,13 @@
|
||||
const RUNTIME_ROLES = new Set(['stable', 'candidate']);
|
||||
|
||||
export function resolvePortalRuntimeRole(env = process.env) {
|
||||
const configured = String(env.MEMIND_RUNTIME_ROLE ?? '').trim().toLowerCase();
|
||||
return RUNTIME_ROLES.has(configured) ? configured : 'stable';
|
||||
}
|
||||
|
||||
export function isPassiveCanaryRuntime(env = process.env) {
|
||||
return (
|
||||
resolvePortalRuntimeRole(env) === 'candidate'
|
||||
&& String(env.MEMIND_CANARY_PASSIVE_RUNTIME ?? '1').trim() !== '0'
|
||||
);
|
||||
}
|
||||
@@ -0,0 +1,30 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
|
||||
import {
|
||||
isPassiveCanaryRuntime,
|
||||
resolvePortalRuntimeRole,
|
||||
} from './portal-runtime-role.mjs';
|
||||
|
||||
test('runtime role defaults to stable and rejects unknown values', () => {
|
||||
assert.equal(resolvePortalRuntimeRole({}), 'stable');
|
||||
assert.equal(resolvePortalRuntimeRole({ MEMIND_RUNTIME_ROLE: 'unknown' }), 'stable');
|
||||
});
|
||||
|
||||
test('candidate runtime is passive unless explicitly disabled', () => {
|
||||
assert.equal(
|
||||
isPassiveCanaryRuntime({ MEMIND_RUNTIME_ROLE: 'candidate' }),
|
||||
true,
|
||||
);
|
||||
assert.equal(
|
||||
isPassiveCanaryRuntime({
|
||||
MEMIND_RUNTIME_ROLE: 'candidate',
|
||||
MEMIND_CANARY_PASSIVE_RUNTIME: '0',
|
||||
}),
|
||||
false,
|
||||
);
|
||||
assert.equal(
|
||||
isPassiveCanaryRuntime({ MEMIND_RUNTIME_ROLE: 'stable' }),
|
||||
false,
|
||||
);
|
||||
});
|
||||
@@ -1,3 +1,5 @@
|
||||
import { resolvePortalRuntimeRole } from './portal-runtime-role.mjs';
|
||||
|
||||
function runtimeEnvFlag(value, fallback = false) {
|
||||
const raw = String(value ?? '').trim().toLowerCase();
|
||||
if (!raw) return fallback;
|
||||
@@ -54,6 +56,7 @@ export function attachPortalRuntimeRoutes(
|
||||
}
|
||||
|
||||
api.get('/status', async (_req, res, next) => {
|
||||
res.setHeader?.('X-Memind-Runtime-Role', resolvePortalRuntimeRole(env));
|
||||
await waitForUserAuthReady();
|
||||
const userAuth = getUserAuth();
|
||||
const tkmindProxy = getTkmindProxy();
|
||||
|
||||
@@ -14,6 +14,7 @@ function createRouterRecorder() {
|
||||
|
||||
function createResponseRecorder() {
|
||||
return {
|
||||
headers: new Map(),
|
||||
statusCode: 200,
|
||||
body: undefined,
|
||||
responseType: null,
|
||||
@@ -31,6 +32,10 @@ function createResponseRecorder() {
|
||||
this.body = body;
|
||||
return this;
|
||||
},
|
||||
setHeader(name, value) {
|
||||
this.headers.set(String(name).toLowerCase(), String(value));
|
||||
return this;
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
@@ -73,6 +78,31 @@ test('GET /status preserves upstream status and body', async () => {
|
||||
assert.equal(res.statusCode, 207);
|
||||
assert.equal(res.responseType, 'send');
|
||||
assert.equal(res.body, 'upstream-status');
|
||||
assert.equal(res.headers.get('x-memind-runtime-role'), 'stable');
|
||||
});
|
||||
|
||||
test('GET /status identifies a passive candidate runtime without changing the body', async () => {
|
||||
const api = createRouterRecorder();
|
||||
attachPortalRuntimeRoutes(api, {
|
||||
env: { MEMIND_RUNTIME_ROLE: 'candidate' },
|
||||
getUserAuth: () => ({}),
|
||||
getTkmindProxy: () => ({
|
||||
async apiFetch() {
|
||||
return {
|
||||
status: 200,
|
||||
async text() {
|
||||
return 'ok';
|
||||
},
|
||||
};
|
||||
},
|
||||
}),
|
||||
});
|
||||
const res = createResponseRecorder();
|
||||
|
||||
await api.routes.get('/status')({}, res, () => {});
|
||||
|
||||
assert.equal(res.body, 'ok');
|
||||
assert.equal(res.headers.get('x-memind-runtime-role'), 'candidate');
|
||||
});
|
||||
|
||||
test('GET /status falls through when multi-user proxy services are unavailable', async () => {
|
||||
|
||||
Reference in New Issue
Block a user