Compare commits

..

4 Commits

Author SHA1 Message Date
john fb6865638a fix: align DeepSeek canary tool rounds with gate
Memind CI / Test, build, and release guards (pull_request) Successful in 1m32s
2026-07-26 23:49:32 +08:00
john 046704816f fix: use container MCP path in portal canary
Memind CI / Test, build, and release guards (push) Successful in 1m41s
2026-07-26 22:34:58 +08:00
john 02f2dc6b17 fix: preserve canary release stdin across edge checks
Memind CI / Test, build, and release guards (push) Successful in 1m35s
2026-07-26 21:24:23 +08:00
john 473ad80882 fix: route portal canary through isolated edge tunnel
Memind CI / Test, build, and release guards (push) Has been cancelled
2026-07-26 20:30:56 +08:00
23 changed files with 900 additions and 136 deletions
+3
View File
@@ -6,6 +6,7 @@ import { Agent, fetch as undiciFetch } from 'undici';
export const DEFAULT_DEEPSEEK_NO_THINK_PORT = 18036;
export const DEFAULT_DEEPSEEK_UPSTREAM = 'https://api.deepseek.com';
export const DEFAULT_MOONSHOT_UPSTREAM = 'https://api.moonshot.cn';
export const DEEPSEEK_COMPAT_CONTRACT_VERSION = 1;
const insecureDispatcher = new Agent({
connect: { rejectUnauthorized: false },
@@ -256,6 +257,8 @@ export function createDeepseekNoThinkProxy({
res.writeHead(200, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({
ok: true,
contractVersion: DEEPSEEK_COMPAT_CONTRACT_VERSION,
deepseekThinking: 'disabled',
upstream: normalizedUpstream,
moonshotUpstream: normalizedMoonshotUpstream,
}));
+21
View File
@@ -1,6 +1,7 @@
import assert from 'node:assert/strict';
import test from 'node:test';
import {
DEEPSEEK_COMPAT_CONTRACT_VERSION,
decodedUpstreamResponseHeaders,
deepseekDisableThinkingEnabled,
flattenLocalJsonSchemaRefs,
@@ -10,6 +11,7 @@ import {
resolveDeepseekNoThinkProxyBaseUrl,
resolveMoonshotCompatProxyBaseUrl,
sanitizeMoonshotToolSchemas,
startDeepseekNoThinkProxy,
} from './deepseek-no-think-proxy.mjs';
test('injectDeepseekThinkingDisabled adds thinking.disabled when absent', () => {
@@ -168,3 +170,22 @@ test('decodedUpstreamResponseHeaders drops stale compression metadata', () => {
'content-type': 'text/event-stream',
});
});
test('compatibility proxy health identifies the enforced production contract', async (t) => {
const server = await startDeepseekNoThinkProxy({
host: '127.0.0.1',
port: 0,
logger: { info() {}, warn() {}, error() {} },
});
t.after(() => new Promise((resolve) => server.close(resolve)));
const address = server.address();
const response = await fetch(`http://127.0.0.1:${address.port}/health`);
assert.equal(response.status, 200);
assert.deepEqual(await response.json(), {
ok: true,
contractVersion: DEEPSEEK_COMPAT_CONTRACT_VERSION,
deepseekThinking: 'disabled',
upstream: 'https://api.deepseek.com',
moonshotUpstream: 'https://api.moonshot.cn',
});
});
+10 -4
View File
@@ -1,6 +1,6 @@
# Local and 103 runtime topology
> Last confirmed: 2026-07-07 09:19 CST.
> Last confirmed: 2026-07-26 20:18 CST.
>
> This is the current topology source of truth for local Memind and 103 production. Prefer this document over older migration notes. Older architecture documents may contain historical paths from before the MindSpace split.
@@ -39,14 +39,20 @@ Portal must be released as a runtime artifact. Do not edit source directly on 10
| Item | Value |
|------|-------|
| Identity router | `127.0.0.1:18080` |
| Stable edge upstream | `105 nginx -> 58.38.22.103:8081` |
| Identity router | `127.0.0.1:18082` |
| Canary edge tunnel | `105 127.0.0.1:19082 -> 103 127.0.0.1:18082` |
| Candidate Portal | `127.0.0.1:18081` |
| Candidate goosed | `127.0.0.1:18015` |
| Release entry | `scripts/release-portal-canary-prod.sh` |
| Rollback entry | `scripts/rollback-portal-canary-prod.sh` |
The stable Portal stays on 8081. The 105 reverse tunnel points to 18080 only after the
candidate, router, backups, immutable identity checks, and fallback checks pass.
The stable Portal stays on 8081 and the normal 105 edge path is a direct upstream, not
the retired 19081 tunnel. A canary release starts an isolated 19082 tunnel and changes
the two committed 105 nginx upstreams only after the candidate, router, 103/105 backups,
immutable identity checks, and tunnel checks pass. It then proves stable fallback before
declaring success. Port 18080 is occupied by a separate Memory V2 mock service and must
not be reused.
## MindSpace Service
+16
View File
@@ -114,6 +114,22 @@ ssh root@120.26.184.105 '
- 禁止在 `105` 直接改 `server.mjs` / `wechat-mp.mjs` / `scripts/wechat-mp-menu.mjs`
- 禁止跳过健康检查就宣称“已发布”
## 103 Portal 用户灰度的 105 入口切换
103 Portal 灰度不发布业务源码到 105。正式入口由已提交的
`scripts/release-portal-canary-prod.sh` 受控变更:
1. 先备份并校验 105 活动的 `m.tkmind.cn.conf``wechat.m.tkmind.cn.conf`
2. 在 103 启动独立路由器 `18082`,并通过反向隧道只暴露为 105 本机 `19082`
3. 候选、身份路由和隧道全部通过后,脚本才把两份 nginx 上游从
`58.38.22.103:8081` 切到 `127.0.0.1:19082`
4. 必须先 `nginx -t`,再 reload,并在切换后主动证明候选故障会回落稳定 8081;
任一步失败恢复备份并回到稳定入口。
5. 回滚只能使用 `scripts/rollback-portal-canary-prod.sh`,禁止在 105 手工 `sed -i`
该动作属于 commit、CI、完整 Gate report 和明确生产批准约束下的正式发布运维,
不构成允许在线编辑 105 配置源码的一般例外。
## 服务号底部菜单(`wechat-mp-menu.mjs`
菜单名称与链接定义在本地:
+3 -1
View File
@@ -545,7 +545,9 @@ failed = skipped = blocked = unknown = cleanup_failed = 0
### 10.0 用户级灰度前置条件
用户级灰度入口由 `scripts/release-portal-canary-prod.sh` 安装:稳定 Portal 保持 8081
候选 Portal 使用 18081,候选 goosed 使用 18015,身份路由器使用 18080。灰度身份必须
候选 Portal 使用 18081,候选 goosed 使用 18015,身份路由器使用 18082105 使用独立
19082 反向隧道访问路由器。活动 nginx 上游在脚本备份、`nginx -t` 和健康检查后,才允许
从稳定 `58.38.22.103:8081` 切到 `127.0.0.1:19082`。灰度身份必须
使用不可变用户 ID、唯一用户名或微信绑定账号标识;禁止按显示昵称(包括“唐”)匹配。未命中、
身份解析失败或候选不可用必须自动回落稳定版本。
+37 -12
View File
@@ -6,15 +6,36 @@ Portal 的首个生产动作必须是用户级灰度,不能直接整包替换
| 组件 | 地址 | 说明 |
|---|---|---|
| 稳定 Portal | `127.0.0.1:8081` | 保持当前 live 目录和进程,不在灰度发布替换 |
| 灰度路由器 | `127.0.0.1:18080` | 105 的 `19081` 反向隧道在全部检查通过后切到这里 |
| 稳定 Portal | `103:8081` | 105 当前直接回源 `58.38.22.103:8081`灰度发布替换该进程 |
| 灰度路由器 | `103 127.0.0.1:18082` | 避开已占用的 18080;只通过专用灰度隧道暴露给 105 |
| 灰度边缘隧道 | `105 127.0.0.1:19082` | 反向 SSH 到 103 的 18082;不复用已退役的 19081 链路 |
| 候选 Portal | `127.0.0.1:18081` | 独立、无源码 runtime;使用被动运行时模式 |
| 候选 goosed | `127.0.0.1:18015` | 独立容器并挂载候选 runtime,不重建稳定 9 个容器 |
| DeepSeek 兼容代理 | `103 0.0.0.0:18036` | 仅供宿主机和候选容器使用;不接入 105 或公网入口 |
候选 Portal 通过 `MEMIND_PORTAL_H5_ROOT` 共享经审计的持久数据和数据库,但
`MEMIND_RUNTIME_ROLE=candidate` 会禁止重复启动 MindSpace、Plaza、提醒、订阅续期和
Agent 恢复扫描等单例后台任务。
候选 Portal 必须从候选 runtime 目录启动,使相对模块只解析到本次候选 artifact。
候选 goosed 在 Docker 内执行扩展,MCP Node 和入口路径必须固定为
`/usr/local/bin/node``/opt/portal/mindspace-sandbox-mcp.mjs`;禁止继承稳定
`.env` 中仅宿主机可见的 `/Users/...` 路径。若日志出现
`Failed to add extension` 和宿主机绝对路径的 `MODULE_NOT_FOUND`,应先核对候选
进程环境与 `/opt/portal` 容器挂载,禁止把宿主机文件复制进容器临时修补。
DeepSeek V4 的工具轮次还必须经过候选 runtime 中的
`deepseek-no-think-proxy.mjs`。候选启动脚本必须强制
`MEMIND_DEEPSEEK_DISABLE_THINKING=1`,并把 goosed 自定义 provider 指向
`http://host.docker.internal:18036/v1`。该代理通过独立 LaunchAgent 管理;
发布必须同时验证 `/health` 的兼容契约和候选容器到该地址的连通性。灰度路由器把
该健康地址纳入候选健康闭包,代理掉线时必须自动回落稳定 Portal。
若日志出现
`The reasoning_content in the thinking mode must be passed back to the API.`
说明 DeepSeek 请求绕过了兼容契约。禁止只在线补环境变量:必须检查 artifact、
LaunchAgent、Portal provider 同步和路由器依赖健康配置是否属于同一 release。
## 身份和回落规则
- `john`:按数据库中的唯一用户名或用户 ID 匹配。
@@ -24,6 +45,7 @@ Agent 恢复扫描等单例后台任务。
- 微信回调通过 `FromUserName``h5_user_wechat_identities` 只读解析到绑定账号,再用唯一账号标识匹配。
- 未命中、无身份、过期/撤销会话、数据库异常:走稳定版本。
- 候选 `/api/status` 不健康:自动走稳定版本。
- DeepSeek 兼容代理不健康:自动走稳定版本。
- SSE、附件、公开页和普通 API 均由同一反向代理转发;微信 XML body 必须原样保留。
## 发布入口
@@ -40,14 +62,17 @@ bash scripts/release-portal-canary-prod.sh --yes
1. 验证源码、分支、远端主线、Gate report 和 artifact hash。
2. 执行 103 只读预检。
3. 上传经过验证的候选包。
4. 创建稳定 live 全量备份持久目录备份,并校验 SHA256、gzip 和关键目录
5. 启动独立候选 goosed、候选 Portal 和灰度路由器
6. 验证不可变身份配置、稳定/候选健康和路由器诊断
7. 短暂阻止新 Agent Run,排空活动任务
8. 只把反向隧道从 8081 切到 18080
9. 主动停止候选,证明目标身份会自动回落稳定版,再恢复候选
4. 创建稳定 live 全量备份持久目录备份和 105 两份活动 nginx 配置备份,并校验完整性
5. 启动独立候选 goosed 和 DeepSeek 兼容代理,验证兼容契约及容器连通性
6. 启动候选 Portal 和灰度路由器。
7. 验证不可变身份配置、稳定/候选/依赖健康和路由器诊断
8. 启动独立的 `105:19082 -> 103:18082` 灰度隧道并从 105 验证
9. 短暂阻止新 Agent Run,排空活动任务
10. 逐文件原子替换、验证并 reload 105 nginx,把活动上游从 `58.38.22.103:8081` 切到 `127.0.0.1:19082`
11. 主动停止候选,证明目标身份会自动回落稳定版,再恢复候选。
任一步失败都会把隧道恢复到稳定 8081,停止候选服务并保留候选目录和备份证据。
任一步失败都会把 105 nginx 恢复到稳定 `58.38.22.103:8081`,停止灰度隧道和候选服务,
并保留候选目录与备份证据。活动 nginx 配置变更只能由已提交的发布脚本执行,禁止在线手改。
## 回滚
@@ -57,10 +82,10 @@ bash scripts/rollback-portal-canary-prod.sh --yes
回滚只执行以下动作:
- 把 105 反向隧道恢复到稳定 Portal 8081
- 停止灰度路由器、候选 Portal候选 goosed。
- 把 105 两份活动 nginx 上游恢复到稳定 Portal `58.38.22.103:8081` 并验证 reload
- 停止灰度隧道、灰度路由器、候选 Portal候选 goosed 和 DeepSeek 兼容代理
- 清除发布排水标记。
- 验证稳定 Portal 和 105 隧道健康。
- 验证稳定 Portal 和 105 HTTPS 入口健康。
不会删除候选目录、备份或稳定运行目录。
+38 -4
View File
@@ -61,17 +61,20 @@ bash scripts/release-portal-canary-prod.sh --yes
- 上传 artifact 到 103。
- 103 备份当前 `/Users/john/Project/Memind` 全目录。
- 103 单独备份持久目录:`.env``MindSpace/``data/``users/``.tailscale/``public/plaza-covers/``logs/`
- 原子切换 live 目录
- 更新 LaunchAgent 指向 `/Users/john/Project/Memind/scripts/run-memind-portal-prod.sh`
- 健康检查 `http://127.0.0.1:8081/api/status` 返回 200。
- 备份并校验 105 的 `m.tkmind.cn.conf``wechat.m.tkmind.cn.conf`
- 保持稳定 Portal `103:8081` 不变,启动候选 Portal `18081`、候选 goosed `18015` 和身份路由器 `18082`
- 启动独立 `105:19082 -> 103:18082` 灰度隧道;禁止复用历史 19081 或占用 18080。
- `nginx -t` 通过后才把 105 活动上游从 `58.38.22.103:8081` 切到 `127.0.0.1:19082`
- 健康检查稳定、候选、回落和 105 HTTPS 入口均通过。
- 校验 `MindSpace/*/public/*.html` 下载/附件相对链接均已落盘(`check-mindspace-public-links.mjs --downloads-only`)。
发布成功后,记录脚本输出里的:
- `release_id`
- `archived_source`
- `candidate_dir`
- `full_backup`
- `persist_backup`
- 105 nginx 配置备份
- `git_head`
## 3. Portal 发布后必须验收
@@ -207,6 +210,37 @@ Provider 'custom_deepseek' is not configured
注意:DeepSeek API key / provider secret 是运行配置,不应该烘焙进镜像。
### 4.7 DeepSeek 工具轮次兼容必须与生产拓扑一致
DeepSeek V4 思考模式在工具调用后的下一轮要求客户端回传上一轮
`reasoning_content`。当前 goosed 工具链不能保证完整回传,因此生产候选必须使用
artifact 内的 `deepseek-no-think-proxy.mjs`,由独立 LaunchAgent 在 `18036`
启动,并强制候选 Portal 设置:
```bash
MEMIND_DEEPSEEK_DISABLE_THINKING=1
MEMIND_DEEPSEEK_NO_THINK_PORT=18036
MEMIND_GOOSED_HOST_GATEWAY=host.docker.internal
```
发布不能只看 Portal `/api/status`。还必须验证:
- `http://127.0.0.1:18036/health` 返回兼容契约版本和
`deepseekThinking=disabled`
- `goosed-prod-canary` 能访问
`http://host.docker.internal:18036/health`
- 灰度路由器把该地址纳入候选健康闭包;
- 失败清理和回滚会停止对应 LaunchAgent。
Gate 本地栈和生产候选必须使用同一组开关与进程依赖。若出现:
```text
The reasoning_content in the thinking mode must be passed back to the API.
```
说明实际生产请求绕过了兼容契约;即使场景 Gate 全部通过,也必须阻断发布并排查
环境漂移。
## 5. goosed-prod 镜像更新必看
当前已验证镜像:
+3 -3
View File
@@ -800,13 +800,13 @@ test('syncProfileToGoosed writes provider, model and secret keys for builtin', a
);
});
test('syncProfileToGoosed routes local DeepSeek through the no-thinking proxy', async () => {
test('syncProfileToGoosed routes explicitly protected production DeepSeek through the compatibility proxy', async () => {
const previousProfile = process.env.MEMIND_RUNTIME_PROFILE;
const previousDisableThinking = process.env.MEMIND_DEEPSEEK_DISABLE_THINKING;
const previousProxyBase = process.env.MEMIND_DEEPSEEK_NO_THINK_BASE_URL;
const previousGateway = process.env.MEMIND_GOOSED_HOST_GATEWAY;
process.env.MEMIND_RUNTIME_PROFILE = 'local';
delete process.env.MEMIND_DEEPSEEK_DISABLE_THINKING;
process.env.MEMIND_RUNTIME_PROFILE = 'production';
process.env.MEMIND_DEEPSEEK_DISABLE_THINKING = '1';
delete process.env.MEMIND_DEEPSEEK_NO_THINK_BASE_URL;
process.env.MEMIND_GOOSED_HOST_GATEWAY = 'host.lima.internal';
+47 -3
View File
@@ -244,9 +244,43 @@ function appendForwardedFor(current, remoteAddress) {
return values.join(', ');
}
function dependencyHealthRequest(target, timeoutMs) {
return new Promise((resolve) => {
const request = http.request(
{
hostname: target.hostname,
port: target.port,
method: 'GET',
path: `${target.pathname}${target.search}`,
headers: { connection: 'close' },
},
(response) => {
response.resume();
response.once('end', () => resolve(response.statusCode === 200));
},
);
request.setTimeout(timeoutMs, () => request.destroy());
request.once('error', () => resolve(false));
request.end();
});
}
function normalizeLoopbackHealthTarget(value, label) {
const target = new URL(String(value ?? ''));
if (
target.protocol !== 'http:'
|| !['127.0.0.1', 'localhost', '::1'].includes(target.hostname)
) {
throw new Error(`${label} must be a loopback http URL`);
}
target.hash = '';
return target;
}
export function createMemindCanaryProxy({
stableTarget,
candidateTarget,
candidateHealthDependencies = [],
policy,
identityResolver,
diagnosticSecret,
@@ -256,6 +290,8 @@ export function createMemindCanaryProxy({
} = {}) {
const stable = normalizeLoopbackTarget(stableTarget, 'stable target');
const candidate = normalizeLoopbackTarget(candidateTarget, 'candidate target');
const candidateDependencies = candidateHealthDependencies.map((value, index) =>
normalizeLoopbackHealthTarget(value, `candidate health dependency ${index + 1}`));
if (!policy) throw new Error('canary proxy requires a routing policy');
if (!identityResolver) throw new Error('canary proxy requires an identity resolver');
if (!diagnosticSecret) throw new Error('canary proxy requires a diagnostic secret');
@@ -269,8 +305,13 @@ export function createMemindCanaryProxy({
return healthCache.healthy;
}
if (!healthPromise) {
healthPromise = healthRequest(candidate, candidateHealthTimeoutMs, 'candidate')
.then((healthy) => {
healthPromise = Promise.all([
healthRequest(candidate, candidateHealthTimeoutMs, 'candidate'),
...candidateDependencies.map((target) =>
dependencyHealthRequest(target, candidateHealthTimeoutMs)),
])
.then((results) => {
const healthy = results.every(Boolean);
healthCache = { checkedAt: Date.now(), healthy };
return healthy;
})
@@ -534,11 +575,14 @@ async function main() {
|| '',
});
const listenHost = process.env.MEMIND_CANARY_PROXY_HOST || '127.0.0.1';
const listenPort = Number(process.env.MEMIND_CANARY_PROXY_PORT ?? 18080);
const listenPort = Number(process.env.MEMIND_CANARY_PROXY_PORT ?? 18082);
const proxy = createMemindCanaryProxy({
stableTarget: process.env.MEMIND_CANARY_STABLE_URL || 'http://127.0.0.1:8081',
candidateTarget:
process.env.MEMIND_CANARY_CANDIDATE_URL || 'http://127.0.0.1:18081',
candidateHealthDependencies: csvValues(
process.env.MEMIND_CANARY_CANDIDATE_HEALTH_URLS,
),
policy,
identityResolver,
diagnosticSecret: process.env.MEMIND_CANARY_DIAGNOSTIC_SECRET,
+48
View File
@@ -287,6 +287,54 @@ test('identity lookup errors fail closed to stable', async (t) => {
assert.equal(response.headers['x-memind-runtime-route'], 'stable');
});
test('candidate health fails closed when its DeepSeek compatibility dependency is down', async (t) => {
const stableServer = upstream('stable');
const candidateServer = upstream('candidate');
let dependencyHealthy = true;
const dependencyServer = http.createServer((req, res) => {
res.writeHead(req.url === '/health' && dependencyHealthy ? 200 : 503);
res.end();
});
const stableUrl = await listen(stableServer);
const candidateUrl = await listen(candidateServer);
const dependencyUrl = await listen(dependencyServer);
const proxy = createMemindCanaryProxy({
stableTarget: stableUrl,
candidateTarget: candidateUrl,
candidateHealthDependencies: [`${dependencyUrl}/health`],
policy: createCanaryPolicy({ usernames: ['john'] }),
identityResolver: {
async fromSessionToken() {
return { id: 'user-john', username: 'john' };
},
async fromWechatOpenid() {
return null;
},
},
diagnosticSecret: 'diagnostic-secret',
candidateHealthTtlMs: 0,
});
const proxyUrl = await listen(proxy.server);
t.after(async () => {
await close(proxy.server);
await close(stableServer);
await close(candidateServer);
await close(dependencyServer);
});
const candidate = await request(proxyUrl, {
headers: { cookie: 'tkmind_user_session=john-token' },
});
assert.equal(candidate.headers['x-memind-runtime-route'], 'candidate');
dependencyHealthy = false;
const fallback = await request(proxyUrl, {
headers: { cookie: 'tkmind_user_session=john-token' },
});
assert.equal(fallback.body, 'stable:');
assert.equal(fallback.headers['x-memind-runtime-route'], 'stable');
});
test('candidate health requires the explicit candidate runtime role', async (t) => {
const stableServer = upstream('stable');
const wrongRoleServer = upstream('stable');
+2
View File
@@ -6,6 +6,7 @@ import path from 'node:path';
export const REQUIRED_PORTAL_RUNTIME_PATHS = Object.freeze([
'server.mjs',
'memind-canary-proxy.mjs',
'deepseek-no-think-proxy.mjs',
'wechat-mp.bundle.mjs',
'mindspace-sandbox-mcp.mjs',
'tkmind-search-mcp.mjs',
@@ -16,6 +17,7 @@ export const REQUIRED_PORTAL_RUNTIME_PATHS = Object.freeze([
'scripts/run-memind-portal-prod.sh',
'scripts/run-memind-portal-candidate.sh',
'scripts/run-memind-canary-proxy-prod.sh',
'scripts/run-deepseek-compat-proxy-candidate.sh',
'scripts/goosed-canary.compose.yml',
'scripts/check-mindspace-public-links.mjs',
'scripts/load-env.mjs',
@@ -0,0 +1,89 @@
import assert from 'node:assert/strict';
import fs from 'node:fs/promises';
import path from 'node:path';
import test from 'node:test';
import { startDeepseekNoThinkProxy } from '../deepseek-no-think-proxy.mjs';
const ROOT = path.resolve(new URL('..', import.meta.url).pathname);
test('sanitized production tool-round replay reaches DeepSeek with thinking disabled', async (t) => {
const forwarded = [];
const proxy = await startDeepseekNoThinkProxy({
host: '127.0.0.1',
port: 0,
logger: { info() {}, warn() {}, error() {} },
fetchImpl: async (_url, init) => {
forwarded.push(JSON.parse(String(init.body)));
return new Response(JSON.stringify({
choices: [{ message: { role: 'assistant', content: 'synthetic result' } }],
}), {
status: 200,
headers: { 'content-type': 'application/json' },
});
},
});
t.after(() => new Promise((resolve) => proxy.close(resolve)));
const address = proxy.address();
const response = await fetch(
`http://127.0.0.1:${address.port}/v1/chat/completions`,
{
method: 'POST',
headers: {
authorization: 'Bearer synthetic-key',
'content-type': 'application/json',
},
body: JSON.stringify({
model: 'deepseek-v4-pro',
thinking: { type: 'enabled' },
messages: [
{ role: 'user', content: 'Use the synthetic filesystem tool.' },
{ role: 'assistant', tool_calls: [{ id: 'call-1', type: 'function' }] },
{ role: 'tool', tool_call_id: 'call-1', content: 'synthetic file list' },
],
}),
},
);
assert.equal(response.status, 200);
assert.equal(forwarded.length, 1);
assert.deepEqual(forwarded[0].thinking, { type: 'disabled' });
assert.equal(forwarded[0].messages[2].role, 'tool');
});
test('Gate, artifact, candidate routing and rollback share one compatibility contract', async () => {
const [
builder,
localStack,
candidateRunner,
compatRunner,
canaryRelease,
canaryRollback,
] = await Promise.all([
fs.readFile(path.join(ROOT, 'scripts', 'build-portal-runtime.mjs'), 'utf8'),
fs.readFile(path.join(ROOT, 'release-gate', 'local-stack.mjs'), 'utf8'),
fs.readFile(path.join(ROOT, 'scripts', 'run-memind-portal-candidate.sh'), 'utf8'),
fs.readFile(
path.join(ROOT, 'scripts', 'run-deepseek-compat-proxy-candidate.sh'),
'utf8',
),
fs.readFile(path.join(ROOT, 'scripts', 'release-portal-canary-prod.sh'), 'utf8'),
fs.readFile(path.join(ROOT, 'scripts', 'rollback-portal-canary-prod.sh'), 'utf8'),
]);
assert.match(builder, /--outfile=\.runtime\/portal\/deepseek-no-think-proxy\.mjs/);
assert.match(
localStack,
/path\.join\(resolvedPortalRoot, 'deepseek-no-think-proxy\.mjs'\)/,
);
assert.match(candidateRunner, /export MEMIND_DEEPSEEK_DISABLE_THINKING=1/);
assert.match(candidateRunner, /export MEMIND_GOOSED_HOST_GATEWAY=host\.docker\.internal/);
assert.match(compatRunner, /source "\$\{STABLE_ROOT\}\/\.env"/);
assert.match(compatRunner, /export MEMIND_DEEPSEEK_PROXY_ENTRYPOINT=1/);
assert.match(canaryRelease, /run-deepseek-compat-proxy-candidate\.sh/);
assert.match(canaryRelease, /MEMIND_CANARY_CANDIDATE_HEALTH_URLS/);
assert.match(canaryRelease, /host\.docker\.internal:\$\{DEEPSEEK_COMPAT_PORT\}\/health/);
assert.match(canaryRelease, /bootout.*DEEPSEEK_COMPAT_LABEL/);
assert.match(canaryRollback, /bootout.*DEEPSEEK_COMPAT_LABEL/);
});
+1 -1
View File
@@ -380,7 +380,7 @@ export async function createLocalGateStack({
deepseekProxyLogFd = fs.openSync(deepseekProxyLogPath, 'a');
deepseekProxyChild = spawn(
process.execPath,
[path.join(root, 'deepseek-no-think-proxy.mjs')],
[path.join(resolvedPortalRoot, 'deepseek-no-think-proxy.mjs')],
{
cwd: root,
env: {
+226 -8
View File
@@ -1,11 +1,31 @@
import assert from 'node:assert/strict';
import { spawnSync } from 'node:child_process';
import fs from 'node:fs/promises';
import os from 'node:os';
import path from 'node:path';
import test from 'node:test';
const ROOT = path.resolve(new URL('..', import.meta.url).pathname);
const CANARY_RELEASE = path.join(ROOT, 'scripts', 'release-portal-canary-prod.sh');
const CANARY_ROLLBACK = path.join(ROOT, 'scripts', 'rollback-portal-canary-prod.sh');
function heredocBody(source, opener, closer) {
const start = source.indexOf(opener);
assert.ok(start >= 0, `missing heredoc opener: ${opener}`);
const bodyStart = source.indexOf('\n', start) + 1;
const end = source.indexOf(`\n${closer}\n`, bodyStart);
assert.ok(bodyStart > 0 && end >= bodyStart, `missing heredoc closer: ${closer}`);
return source.slice(bodyStart, end);
}
function assertShellParses(source, label) {
const result = spawnSync('bash', ['-n'], {
cwd: ROOT,
encoding: 'utf8',
input: source,
});
assert.equal(result.status, 0, `${label} failed bash -n:\n${result.stderr}`);
}
test('production release verifies gate report before any 103 connection', async () => {
const source = await fs.readFile(
@@ -43,6 +63,16 @@ test('runtime builder materializes and declares required Linux ARM64 native pack
assert.match(source, /缺少原生 \.node 文件/);
});
test('runtime builder bundles the DeepSeek compatibility dependency into the artifact', async () => {
const source = await fs.readFile(
path.join(ROOT, 'scripts', 'build-portal-runtime.mjs'),
'utf8',
);
assert.match(source, /bundleDeepseekCompatProxy/);
assert.match(source, /--outfile=\.runtime\/portal\/deepseek-no-think-proxy\.mjs/);
assert.match(source, /await bundleDeepseekCompatProxy\(\)/);
});
test('packaged runtime gate isolates persistent roots and rejects artifact mutation', async () => {
const localStackSource = await fs.readFile(
path.join(ROOT, 'release-gate', 'local-stack.mjs'),
@@ -50,7 +80,10 @@ test('packaged runtime gate isolates persistent roots and rejects artifact mutat
);
assert.match(localStackSource, /MEMIND_PORTAL_H5_ROOT: sandboxRoot/);
assert.match(localStackSource, /MEMIND_DEEPSEEK_DISABLE_THINKING: '1'/);
assert.match(localStackSource, /deepseek-no-think-proxy\.mjs/);
assert.match(
localStackSource,
/path\.join\(resolvedPortalRoot, 'deepseek-no-think-proxy\.mjs'\)/,
);
assert.match(localStackSource, /MEMIND_ORCHESTRATOR_MODE: 'shadow'/);
assert.match(localStackSource, /MEMIND_ORCHESTRATOR_PAGE_DATA_VALIDATION_GATE_ENABLED: '1'/);
assert.match(localStackSource, /H5_USERS_ROOT: usersRoot/);
@@ -96,28 +129,213 @@ test('production canary keeps stable 8081 live and switches only after verified
const source = await fs.readFile(CANARY_RELEASE, 'utf8');
const fullBackup = source.indexOf('Create and verify the full stable backup');
const persistBackup = source.indexOf('Create and verify the persisted-data backup');
const edgeBackup = source.indexOf('Create and verify the active 105 nginx routing backup');
const goosedStart = source.indexOf('Start an isolated goosed candidate on 18015');
const candidateStart = source.indexOf('Start the passive candidate Portal on 18081');
const proxyStart = source.indexOf('Start the fail-closed identity router on 18080');
const tunnelSwitch = source.indexOf('Switch only the reverse tunnel');
const deepseekCompatStart = source.indexOf(
'Start the DeepSeek tool-round compatibility proxy on 18036',
);
const proxyStart = source.indexOf('Start the fail-closed identity router on 18082');
const tunnelStart = source.indexOf('Start the isolated 105 reverse tunnel on 19082');
const edgeSwitch = source.indexOf(
'Switch the committed 105 nginx upstreams to the isolated canary tunnel',
);
const fallbackProbe = source.indexOf('route-probe?username=john');
assert.ok(fullBackup > 0);
assert.ok(persistBackup > fullBackup);
assert.ok(goosedStart > persistBackup);
assert.ok(candidateStart > goosedStart);
assert.ok(edgeBackup > persistBackup);
assert.ok(goosedStart > edgeBackup);
assert.ok(deepseekCompatStart > goosedStart);
assert.ok(candidateStart > deepseekCompatStart);
assert.ok(proxyStart > candidateStart);
assert.ok(tunnelSwitch > proxyStart);
assert.ok(fallbackProbe > tunnelSwitch);
assert.match(source, /write_tunnel_plist 8081/);
assert.ok(tunnelStart > proxyStart);
assert.ok(edgeSwitch > tunnelStart);
assert.ok(fallbackProbe > edgeSwitch);
assert.match(source, /restore_edge_to_stable/);
assert.match(source, /proxy_pass http:\/\/58\.38\.22\.103:8081;/);
assert.match(source, /proxy_pass http:\/\/127\.0\.0\.1:\$\{CANARY_TUNNEL_REMOTE_PORT\};/);
assert.match(source, /nginx -t/);
assert.match(source, /CANARY_PROXY_PORT=18082/);
assert.match(source, /CANARY_TUNNEL_REMOTE_PORT=19082/);
assert.match(source, /DEEPSEEK_COMPAT_PORT=18036/);
assert.match(source, /deepseek-no-think-proxy\.mjs/);
assert.match(source, /run-deepseek-compat-proxy-candidate\.sh/);
assert.match(source, /MEMIND_CANARY_CANDIDATE_HEALTH_URLS/);
assert.match(source, /deepseek_compat_healthy/);
assert.match(source, /bootout.*DEEPSEEK_COMPAT_LABEL/);
assert.match(
source,
/edge_ssh\(\) \{\s*#.*\n\s*#.*\n\s*ssh -n -o BatchMode=yes/,
);
assert.match(
source,
/ssh -o BatchMode=yes -o ConnectTimeout=10 "\$\{EDGE_HOST\}" \/bin\/bash <<EDGE_SWITCH/,
);
assert.doesNotMatch(source, /edge_ssh \/bin\/bash <<EDGE_SWITCH/);
assert.match(source, /CANARY_USERNAMES="john"/);
assert.match(source, /CANARY_WECHAT_USER_IDS="wx_ul610et8"/);
assert.match(source, /route-probe\?\$\{probe_query\}/);
assert.match(source, /candidate_healthy/);
assert.match(source, /MEMIND_CANARY_RELEASE_ID/);
assert.match(
source,
/if restore_edge_to_stable >\/dev\/null 2>&1; then\s+stop_candidate_services/,
);
assert.match(source, /keeping canary services running/);
assert.match(source, /--resolve wechat\.m\.tkmind\.cn:443:127\.0\.0\.1/);
assert.doesNotMatch(source, /lsof -tiTCP:8081/);
assert.doesNotMatch(source, /bootout.*cn\.tkmind\.memind-portal/);
});
test('candidate runner overrides stable host MCP paths with container-visible paths', async (t) => {
const tempRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'memind-canary-runner-'));
t.after(() => fs.rm(tempRoot, { recursive: true, force: true }));
const stableRoot = path.join(tempRoot, 'stable');
const fakeNode = path.join(tempRoot, 'node');
await fs.mkdir(stableRoot);
await fs.writeFile(
path.join(stableRoot, '.env'),
[
'GOOSED_MCP_NODE_PATH=/opt/homebrew/opt/node@24/bin/node',
'GOOSED_MCP_SERVER_PATH=/Users/john/Project/Memind/mindspace-sandbox-mcp.mjs',
'MEMIND_DEEPSEEK_DISABLE_THINKING=0',
'MEMIND_DEEPSEEK_NO_THINK_PORT=9999',
'MEMIND_DEEPSEEK_NO_THINK_BASE_URL=http://wrong.invalid/v1',
'MEMIND_DEEPSEEK_NO_THINK_HOST=wrong.invalid',
'MEMIND_GOOSED_HOST_GATEWAY=wrong.invalid',
'',
].join('\n'),
);
await fs.writeFile(
fakeNode,
[
'#!/usr/bin/env bash',
'printf "cwd=%s\\n" "$PWD"',
'printf "mcp_node=%s\\n" "$GOOSED_MCP_NODE_PATH"',
'printf "mcp_server=%s\\n" "$GOOSED_MCP_SERVER_PATH"',
'printf "deepseek_disable=%s\\n" "$MEMIND_DEEPSEEK_DISABLE_THINKING"',
'printf "deepseek_port=%s\\n" "$MEMIND_DEEPSEEK_NO_THINK_PORT"',
'printf "deepseek_gateway=%s\\n" "$MEMIND_GOOSED_HOST_GATEWAY"',
'printf "deepseek_base=%s\\n" "${MEMIND_DEEPSEEK_NO_THINK_BASE_URL-unset}"',
'printf "deepseek_host=%s\\n" "${MEMIND_DEEPSEEK_NO_THINK_HOST-unset}"',
'printf "entrypoint=%s\\n" "$1"',
'',
].join('\n'),
{ mode: 0o755 },
);
const result = spawnSync(
'bash',
[path.join(ROOT, 'scripts', 'run-memind-portal-candidate.sh')],
{
cwd: '/',
encoding: 'utf8',
env: {
...process.env,
MEMIND_CANARY_STABLE_ROOT: stableRoot,
MEMIND_CANARY_RELEASE_ID: 'test-release',
MEMIND_CANARY_CANDIDATE_PORT: '65534',
NODE_BIN: fakeNode,
},
},
);
assert.equal(result.status, 0, result.stderr);
assert.match(result.stdout, new RegExp(`^cwd=${ROOT}$`, 'm'));
assert.match(result.stdout, /^mcp_node=\/usr\/local\/bin\/node$/m);
assert.match(
result.stdout,
/^mcp_server=\/opt\/portal\/mindspace-sandbox-mcp\.mjs$/m,
);
assert.match(result.stdout, /^deepseek_disable=1$/m);
assert.match(result.stdout, /^deepseek_port=18036$/m);
assert.match(result.stdout, /^deepseek_gateway=host\.docker\.internal$/m);
assert.match(result.stdout, /^deepseek_base=unset$/m);
assert.match(result.stdout, /^deepseek_host=unset$/m);
assert.match(result.stdout, new RegExp(`^entrypoint=${ROOT}/server\\.mjs$`, 'm'));
});
test('canary rollback stops the DeepSeek compatibility process with the other canary services', async () => {
const source = await fs.readFile(CANARY_ROLLBACK, 'utf8');
assert.match(
source,
/DEEPSEEK_COMPAT_LABEL="cn\.tkmind\.memind-deepseek-compat-candidate"/,
);
assert.match(source, /bootout.*DEEPSEEK_COMPAT_LABEL/);
assert.match(source, /docker rm -f goosed-prod-canary/);
});
test('DeepSeek compatibility runner preserves stable upstream config but enforces candidate controls', async (t) => {
const tempRoot = await fs.mkdtemp(path.join(os.tmpdir(), 'memind-deepseek-runner-'));
t.after(() => fs.rm(tempRoot, { recursive: true, force: true }));
const stableRoot = path.join(tempRoot, 'stable');
const fakeNode = path.join(tempRoot, 'node');
await fs.mkdir(stableRoot);
await fs.writeFile(
path.join(stableRoot, '.env'),
[
'DEEPSEEK_API_BASE_URL=https://synthetic-upstream.invalid/v1',
'MEMIND_DEEPSEEK_PROXY_ENTRYPOINT=0',
'MEMIND_DEEPSEEK_NO_THINK_PORT=9999',
'',
].join('\n'),
);
await fs.writeFile(
fakeNode,
[
'#!/usr/bin/env bash',
'printf "proxy_entrypoint=%s\\n" "$MEMIND_DEEPSEEK_PROXY_ENTRYPOINT"',
'printf "proxy_port=%s\\n" "$MEMIND_DEEPSEEK_NO_THINK_PORT"',
'printf "upstream=%s\\n" "$DEEPSEEK_API_BASE_URL"',
'printf "entrypoint=%s\\n" "$1"',
'',
].join('\n'),
{ mode: 0o755 },
);
const result = spawnSync(
'bash',
[path.join(ROOT, 'scripts', 'run-deepseek-compat-proxy-candidate.sh')],
{
cwd: '/',
encoding: 'utf8',
env: {
...process.env,
MEMIND_CANARY_STABLE_ROOT: stableRoot,
MEMIND_CANARY_DEEPSEEK_PROXY_PORT: '18036',
NODE_BIN: fakeNode,
},
},
);
assert.equal(result.status, 0, result.stderr);
assert.match(result.stdout, /^proxy_entrypoint=1$/m);
assert.match(result.stdout, /^proxy_port=18036$/m);
assert.match(result.stdout, /^upstream=https:\/\/synthetic-upstream\.invalid\/v1$/m);
assert.match(
result.stdout,
new RegExp(`^entrypoint=${ROOT}/deepseek-no-think-proxy\\.mjs$`, 'm'),
);
});
test('canary release and rollback remote shells remain syntactically valid', async () => {
const releaseSource = await fs.readFile(CANARY_RELEASE, 'utf8');
const rollbackSource = await fs.readFile(CANARY_ROLLBACK, 'utf8');
const releaseRemote = heredocBody(releaseSource, "<<'REMOTE_SCRIPT'", 'REMOTE_SCRIPT');
const rollbackRemote = heredocBody(rollbackSource, "<<'REMOTE'", 'REMOTE');
const edgeSwitch = heredocBody(releaseRemote, '<<EDGE_SWITCH', 'EDGE_SWITCH')
.replaceAll('\\$', '$');
const edgeRollback = heredocBody(rollbackRemote, '<<EDGE_ROLLBACK', 'EDGE_ROLLBACK')
.replaceAll('\\$', '$');
assertShellParses(releaseRemote, 'release remote shell');
assertShellParses(rollbackRemote, 'rollback remote shell');
assertShellParses(edgeSwitch, '105 edge switch shell');
assertShellParses(edgeRollback, '105 edge rollback shell');
});
test('release readiness ignores only the generated canary artifact cover path', async () => {
const source = await fs.readFile(
path.join(ROOT, 'scripts', 'check-release-ready.sh'),
@@ -0,0 +1,16 @@
{
"id": "PRC-REL-002",
"scenario_ids": ["REL-06", "REL-09", "REL-11"],
"failure_signature": "gate-production-drift-deepseek-thinking-tool-round-reasoning-content-replay-rejected",
"synthetic_intent": "验证工具调用后的合成 DeepSeek 会话经过生产候选兼容链路,禁用思考模式,并在兼容依赖失效时让灰度流量回落稳定版本。",
"source_batch_id": "incident-20260726-deepseek-tool-round",
"evidence_digest": "97171fe1d94e0297cb0b932cf1ff18e8d1760519300f7308b9f8cc4e9517e022",
"contains_personal_data": false,
"privacy_review": {
"reviewed_by": "release-gate-maintainers",
"reviewed_at": "2026-07-26T23:55:00+08:00"
},
"automation": {
"command": ["node", "--test", "release-gate/deepseek-production-parity.test.mjs"]
}
}
@@ -1,6 +1,6 @@
{
"fixture_ids": ["PRC-REL-001"],
"fixture_ids": ["PRC-REL-001", "PRC-REL-002"],
"reviewed_by": "release-gate-maintainers",
"reviewed_at": "2026-07-26T12:00:00+08:00",
"notes": "仅收录已脱敏的 runtime 启动回归签名;不包含聊天原文、用户身份或生产地址。"
"notes": "仅收录已脱敏的 runtime 启动和生产配置同构回归签名;不包含聊天原文、用户身份或生产地址。"
}
+27
View File
@@ -156,6 +156,26 @@ async function bundleCanaryProxy() {
await run(esbuildBin, args);
}
async function bundleDeepseekCompatProxy() {
if (!(await exists(esbuildBin))) {
throw new Error(`未找到 esbuild: ${esbuildBin}`);
}
console.log('==> 打包 DeepSeek 工具轮次兼容代理');
const args = [
'deepseek-no-think-proxy.mjs',
'--bundle',
'--platform=node',
'--format=esm',
`--target=${runtimeNodeTarget}`,
'--outfile=.runtime/portal/deepseek-no-think-proxy.mjs',
'--banner:js=import { createRequire as __createRequire } from "node:module"; const require = __createRequire(import.meta.url);',
];
for (const pkg of externalPackages) {
args.push(`--external:${pkg}`);
}
await run(esbuildBin, args);
}
async function bundleSandboxMcp() {
if (!(await exists(esbuildBin))) {
throw new Error(`未找到 esbuild: ${esbuildBin}`);
@@ -409,6 +429,7 @@ async function writeMetadata() {
for (const scriptName of [
'run-memind-portal-candidate.sh',
'run-memind-canary-proxy-prod.sh',
'run-deepseek-compat-proxy-candidate.sh',
]) {
const source = path.join(root, 'scripts', scriptName);
if (!(await exists(source))) {
@@ -527,6 +548,7 @@ async function writeMetadata() {
' tkmind-excel-mcp.mjs (esbuild bundle; required when Excel Analyst is enabled)',
' wechat-mp.bundle.mjs (esbuild bundle; hot-swappable WeChat MP module)',
' memind-canary-proxy.mjs (immutable-identity stable/candidate router)',
' deepseek-no-think-proxy.mjs (DeepSeek V4 tool-round compatibility proxy)',
'',
'Post-deploy validation (scripts/check-mindspace-public-links.mjs):',
' Scans MindSpace/*/public/*.html for missing relative href/src/cover targets.',
@@ -598,6 +620,7 @@ async function main() {
await copyRuntimeAssets();
await bundleServer();
await bundleCanaryProxy();
await bundleDeepseekCompatProxy();
await bundleWechatMp();
await bundleSandboxMcp();
await bundleMindSearchMcp();
@@ -611,6 +634,10 @@ async function main() {
await fs.chmod(path.join(runtimeRoot, 'scripts', 'run-memind-portal-prod.sh'), 0o755);
await fs.chmod(path.join(runtimeRoot, 'scripts', 'run-memind-portal-candidate.sh'), 0o755);
await fs.chmod(path.join(runtimeRoot, 'scripts', 'run-memind-canary-proxy-prod.sh'), 0o755);
await fs.chmod(
path.join(runtimeRoot, 'scripts', 'run-deepseek-compat-proxy-candidate.sh'),
0o755,
);
await fs.chmod(path.join(runtimeRoot, 'scripts', 'wechat-mp-menu.mjs'), 0o755);
await fs.chmod(path.join(runtimeRoot, 'scripts', 'check-stream-runtime.mjs'), 0o755);
await fs.chmod(path.join(runtimeRoot, 'scripts', 'runtime-worker-drain.mjs'), 0o755);
+225 -55
View File
@@ -12,9 +12,14 @@ GOOSED_DIR="${REMOTE_ROOT}/goosed-prod"
RUNTIME_ROOT="${ROOT}/.runtime/portal"
PORTAL_CANDIDATE_LABEL="cn.tkmind.memind-portal-candidate"
CANARY_PROXY_LABEL="cn.tkmind.memind-canary-proxy"
PORTAL_TUNNEL_LABEL="cn.tkmind.memind-portal-tunnel"
MEMIND_PORTAL_TUNNEL_HOST="${MEMIND_PORTAL_TUNNEL_HOST:-ssh105-public}"
MEMIND_PORTAL_TUNNEL_REMOTE_PORT="${MEMIND_PORTAL_TUNNEL_REMOTE_PORT:-19081}"
CANARY_TUNNEL_LABEL="cn.tkmind.memind-canary-tunnel"
DEEPSEEK_COMPAT_LABEL="cn.tkmind.memind-deepseek-compat-candidate"
EDGE_HOST="${MEMIND_CANARY_EDGE_HOST:-ssh105-public}"
CANARY_PROXY_PORT=18082
CANARY_TUNNEL_REMOTE_PORT=19082
DEEPSEEK_COMPAT_PORT=18036
EDGE_MOBILE_CONFIG="/etc/nginx/conf.d/m.tkmind.cn.conf"
EDGE_WECHAT_CONFIG="/etc/nginx/conf.d/wechat.m.tkmind.cn.conf"
CANARY_USERNAMES="john"
CANARY_WECHAT_USER_IDS="wx_ul610et8"
RELEASE_TS="$(date +%Y%m%d-%H%M%S)"
@@ -40,13 +45,15 @@ Usage:
Deploys a candidate Portal beside the stable 103 runtime:
stable Portal 127.0.0.1:8081
canary router 127.0.0.1:18080
canary router 127.0.0.1:18082
candidate Portal 127.0.0.1:18081
candidate goosed 127.0.0.1:18015
DeepSeek compat 0.0.0.0:18036 (host/container only)
The stable runtime is not replaced. The 105 reverse tunnel moves to the canary
router only after backups, candidate health, immutable identity checks, and
automatic fallback checks pass.
The stable runtime is not replaced. A dedicated reverse tunnel exposes the
router only to 105 at 127.0.0.1:19082. The committed release workflow updates
the two active 105 nginx upstreams only after backups, candidate health, and
immutable identity checks pass, then proves automatic fallback before success.
EOF
}
@@ -119,6 +126,7 @@ say "Run release source guards"
required_runtime_paths=(
server.mjs
memind-canary-proxy.mjs
deepseek-no-think-proxy.mjs
wechat-mp.bundle.mjs
mindspace-sandbox-mcp.mjs
tkmind-search-mcp.mjs
@@ -127,6 +135,7 @@ required_runtime_paths=(
package.json
scripts/run-memind-portal-candidate.sh
scripts/run-memind-canary-proxy-prod.sh
scripts/run-deepseek-compat-proxy-candidate.sh
scripts/goosed-canary.compose.yml
scripts/memind-portal-tunnel.sh
)
@@ -151,7 +160,7 @@ printf '%s %s\n' "${bundle_sha}" "$(basename "${BUNDLE_PATH}")" > "${SHA_PATH}"
echo "git_branch=${branch}"
echo "artifact_tree=.runtime/portal"
echo "artifact_bundle_sha256=${bundle_sha}"
echo "routing=stable:8081,proxy:18080,candidate:18081,goosed-canary:18015"
echo "routing=stable:8081,proxy:18082,edge-tunnel:19082,candidate:18081,goosed-canary:18015,deepseek-compat:18036"
echo "canary_usernames=${CANARY_USERNAMES}"
echo "canary_wechat_user_ids=${CANARY_WECHAT_USER_IDS}"
echo "manual_environment_changes=none"
@@ -169,15 +178,29 @@ ssh -o BatchMode=yes -o ConnectTimeout=15 "${HOST}" \
&& test -f '${STABLE_DIR}/.env' \
&& test -f '${GOOSED_DIR}/docker-compose.prod.yml' \
&& curl -fsS http://127.0.0.1:8081/api/status >/dev/null \
&& ! lsof -nP -iTCP:${CANARY_PROXY_PORT} -sTCP:LISTEN >/dev/null 2>&1 \
&& ! lsof -nP -iTCP:${DEEPSEEK_COMPAT_PORT} -sTCP:LISTEN >/dev/null 2>&1 \
&& ! lsof -nP -iTCP:18081 -sTCP:LISTEN >/dev/null 2>&1 \
&& ! lsof -nP -iTCP:18015 -sTCP:LISTEN >/dev/null 2>&1 \
&& /opt/homebrew/bin/docker inspect goosed-prod-1 >/dev/null \
&& test \"\$(df -Pk '${REMOTE_ROOT}' | awk 'NR==2 {print \$4}')\" -gt 10485760"
&& test \"\$(df -Pk '${REMOTE_ROOT}' | awk 'NR==2 {print \$4}')\" -gt 10485760 \
&& ssh -o BatchMode=yes -o ConnectTimeout=10 '${EDGE_HOST}' \
\"test -f '${EDGE_MOBILE_CONFIG}' \
&& test -f '${EDGE_WECHAT_CONFIG}' \
&& grep -q 'proxy_pass http://58.38.22.103:8081;' '${EDGE_MOBILE_CONFIG}' \
&& grep -q 'proxy_pass http://58.38.22.103:8081;' '${EDGE_WECHAT_CONFIG}' \
&& ! grep -q 'proxy_pass http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT};' '${EDGE_MOBILE_CONFIG}' \
&& ! grep -q 'proxy_pass http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT};' '${EDGE_WECHAT_CONFIG}' \
&& nginx -t >/dev/null 2>&1 \
&& ! lsof -nP -iTCP:${CANARY_TUNNEL_REMOTE_PORT} -sTCP:LISTEN >/dev/null 2>&1\""
if [[ "${AUTO_YES}" -ne 1 ]]; then
say "Production canary confirmation"
echo "Target host: ${HOST}"
echo "Release: ${RELEASE_ID}"
echo "Commit: ${FULL_SHA}"
echo "Stable remains on 8081; selected immutable identities route to candidate 18081."
echo "Stable remains on 8081; 105 will use the isolated 19082 canary tunnel."
echo "Selected immutable identities route to candidate 18081."
read -r -p "Continue with the 103 canary release? [y/N] " confirm </dev/tty
[[ "${confirm}" =~ ^[Yy]$ ]] || exit 0
fi
@@ -196,9 +219,14 @@ ssh -o BatchMode=yes "${HOST}" \
GOOSED_DIR='${GOOSED_DIR}' \
PORTAL_CANDIDATE_LABEL='${PORTAL_CANDIDATE_LABEL}' \
CANARY_PROXY_LABEL='${CANARY_PROXY_LABEL}' \
PORTAL_TUNNEL_LABEL='${PORTAL_TUNNEL_LABEL}' \
MEMIND_PORTAL_TUNNEL_HOST='${MEMIND_PORTAL_TUNNEL_HOST}' \
MEMIND_PORTAL_TUNNEL_REMOTE_PORT='${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}' \
CANARY_TUNNEL_LABEL='${CANARY_TUNNEL_LABEL}' \
DEEPSEEK_COMPAT_LABEL='${DEEPSEEK_COMPAT_LABEL}' \
EDGE_HOST='${EDGE_HOST}' \
CANARY_PROXY_PORT='${CANARY_PROXY_PORT}' \
CANARY_TUNNEL_REMOTE_PORT='${CANARY_TUNNEL_REMOTE_PORT}' \
DEEPSEEK_COMPAT_PORT='${DEEPSEEK_COMPAT_PORT}' \
EDGE_MOBILE_CONFIG='${EDGE_MOBILE_CONFIG}' \
EDGE_WECHAT_CONFIG='${EDGE_WECHAT_CONFIG}' \
CANARY_USERNAMES='${CANARY_USERNAMES}' \
CANARY_WECHAT_USER_IDS='${CANARY_WECHAT_USER_IDS}' \
/bin/bash" <<'REMOTE_SCRIPT'
@@ -211,13 +239,15 @@ MANIFEST="${INCOMING_DIR}/memind-portal-canary-${RELEASE_ID}.manifest.txt"
SHA_FILE="${INCOMING_DIR}/memind-portal-canary-${RELEASE_ID}.sha256"
FULL_BACKUP_TAR="${BACKUP_DIR}/memind-full-${RELEASE_ID}-before-canary.tar.gz"
PERSIST_BACKUP_TAR="${BACKUP_DIR}/memind-persisted-${RELEASE_ID}-before-canary.tar.gz"
TUNNEL_PLIST="${HOME}/Library/LaunchAgents/${PORTAL_TUNNEL_LABEL}.plist"
TUNNEL_PLIST_BACKUP="${BACKUP_DIR}/${PORTAL_TUNNEL_LABEL}-${RELEASE_ID}.plist"
CANDIDATE_PLIST="${HOME}/Library/LaunchAgents/${PORTAL_CANDIDATE_LABEL}.plist"
PROXY_PLIST="${HOME}/Library/LaunchAgents/${CANARY_PROXY_LABEL}.plist"
CANARY_TUNNEL_PLIST="${HOME}/Library/LaunchAgents/${CANARY_TUNNEL_LABEL}.plist"
DEEPSEEK_COMPAT_PLIST="${HOME}/Library/LaunchAgents/${DEEPSEEK_COMPAT_LABEL}.plist"
SECRET_FILE="${HOME}/.config/memind/canary-router.secret"
DOCKER_BIN="/opt/homebrew/bin/docker"
GOOSED_COMPOSE="${GOOSED_DIR}/docker-compose.prod.yml"
EDGE_MOBILE_BACKUP="${EDGE_MOBILE_CONFIG}.before-canary-${RELEASE_ID}"
EDGE_WECHAT_BACKUP="${EDGE_WECHAT_CONFIG}.before-canary-${RELEASE_ID}"
say() {
printf '\n[remote %s] %s\n' "$(date +%H:%M:%S)" "$*"
@@ -239,48 +269,103 @@ candidate_healthy() {
return 1
}
write_tunnel_plist() {
local local_port="$1"
cat > "${TUNNEL_PLIST}" <<EOF
deepseek_compat_healthy() {
curl -fsS --max-time 5 "http://127.0.0.1:${DEEPSEEK_COMPAT_PORT}/health" \
| /opt/homebrew/opt/node@24/bin/node --input-type=module -e '
let input = "";
process.stdin.on("data", (chunk) => { input += chunk; });
process.stdin.on("end", () => {
const status = JSON.parse(input);
if (
status.ok !== true
|| status.contractVersion !== 1
|| status.deepseekThinking !== "disabled"
) process.exit(1);
});
'
}
edge_ssh() {
# This remote release script itself arrives on stdin. Ordinary nested SSH
# commands must not consume the unparsed remainder of that script.
ssh -n -o BatchMode=yes -o ConnectTimeout=10 "${EDGE_HOST}" "$@"
}
write_canary_tunnel_plist() {
cat > "${CANARY_TUNNEL_PLIST}" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key><string>${PORTAL_TUNNEL_LABEL}</string>
<key>Label</key><string>${CANARY_TUNNEL_LABEL}</string>
<key>ProgramArguments</key>
<array><string>${STABLE_DIR}/scripts/memind-portal-tunnel.sh</string></array>
<key>EnvironmentVariables</key>
<dict>
<key>MEMIND_PORTAL_TUNNEL_HOST</key><string>${MEMIND_PORTAL_TUNNEL_HOST}</string>
<key>MEMIND_PORTAL_TUNNEL_LOCAL_PORT</key><string>${local_port}</string>
<key>MEMIND_PORTAL_TUNNEL_REMOTE_PORT</key><string>${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}</string>
<key>MEMIND_PORTAL_TUNNEL_HOST</key><string>${EDGE_HOST}</string>
<key>MEMIND_PORTAL_TUNNEL_LOCAL_PORT</key><string>${CANARY_PROXY_PORT}</string>
<key>MEMIND_PORTAL_TUNNEL_REMOTE_PORT</key><string>${CANARY_TUNNEL_REMOTE_PORT}</string>
</dict>
<key>RunAtLoad</key><true/>
<key>KeepAlive</key><true/>
<key>StandardOutPath</key><string>${HOME}/Library/Logs/memind-portal-tunnel.log</string>
<key>StandardErrorPath</key><string>${HOME}/Library/Logs/memind-portal-tunnel.log</string>
<key>StandardOutPath</key><string>${HOME}/Library/Logs/memind-canary-tunnel.log</string>
<key>StandardErrorPath</key><string>${HOME}/Library/Logs/memind-canary-tunnel.log</string>
</dict>
</plist>
EOF
launchctl bootout "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null 2>&1 || true
launchctl bootstrap "${LAUNCHD_GUI}" "${TUNNEL_PLIST}" >/dev/null
launchctl enable "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null 2>&1 || true
launchctl kickstart -k "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null
launchctl bootout "${LAUNCHD_GUI}/${CANARY_TUNNEL_LABEL}" >/dev/null 2>&1 || true
launchctl bootstrap "${LAUNCHD_GUI}" "${CANARY_TUNNEL_PLIST}" >/dev/null
launchctl enable "${LAUNCHD_GUI}/${CANARY_TUNNEL_LABEL}" >/dev/null 2>&1 || true
launchctl kickstart -k "${LAUNCHD_GUI}/${CANARY_TUNNEL_LABEL}" >/dev/null
}
restore_edge_to_stable() {
edge_ssh \
"set -euo pipefail
mobile_backup_exists=0
wechat_backup_exists=0
[[ -f '${EDGE_MOBILE_BACKUP}' ]] && mobile_backup_exists=1
[[ -f '${EDGE_WECHAT_BACKUP}' ]] && wechat_backup_exists=1
[[ \"\${mobile_backup_exists}\" == \"\${wechat_backup_exists}\" ]]
if [[ \"\${mobile_backup_exists}\" -eq 1 ]]; then
test -f '${EDGE_MOBILE_BACKUP}.sha256'
sha256sum -c '${EDGE_MOBILE_BACKUP}.sha256' >/dev/null
cp '${EDGE_MOBILE_BACKUP}' '${EDGE_MOBILE_CONFIG}'
cp '${EDGE_WECHAT_BACKUP}' '${EDGE_WECHAT_CONFIG}'
fi
grep -q 'proxy_pass http://58.38.22.103:8081;' '${EDGE_MOBILE_CONFIG}'
grep -q 'proxy_pass http://58.38.22.103:8081;' '${EDGE_WECHAT_CONFIG}'
! grep -q 'proxy_pass http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT};' '${EDGE_MOBILE_CONFIG}'
! grep -q 'proxy_pass http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT};' '${EDGE_WECHAT_CONFIG}'
nginx -t >/dev/null
systemctl reload nginx
curl -kfsS --max-time 15 \
--resolve m.tkmind.cn:443:127.0.0.1 \
https://m.tkmind.cn/api/status >/dev/null
curl -kfsS --max-time 15 \
--resolve wechat.m.tkmind.cn:443:127.0.0.1 \
https://wechat.m.tkmind.cn/api/status >/dev/null"
}
stop_candidate_services() {
launchctl bootout "${LAUNCHD_GUI}/${CANARY_TUNNEL_LABEL}" >/dev/null 2>&1 || true
launchctl bootout "${LAUNCHD_GUI}/${CANARY_PROXY_LABEL}" >/dev/null 2>&1 || true
launchctl bootout "${LAUNCHD_GUI}/${PORTAL_CANDIDATE_LABEL}" >/dev/null 2>&1 || true
launchctl bootout "${LAUNCHD_GUI}/${DEEPSEEK_COMPAT_LABEL}" >/dev/null 2>&1 || true
"${DOCKER_BIN}" rm -f goosed-prod-canary >/dev/null 2>&1 || true
}
rollback() {
local status=$?
trap - ERR
say "Canary rollout failed; restoring stable-only tunnel"
say "Canary rollout failed; restoring the direct stable edge upstream"
rm -f "${STABLE_DIR}/.release-drain"
write_tunnel_plist 8081 >/dev/null 2>&1 || true
stop_candidate_services
if restore_edge_to_stable >/dev/null 2>&1; then
stop_candidate_services
else
say "CRITICAL: edge restore could not be verified; keeping canary services running"
say "Run scripts/rollback-portal-canary-prod.sh after checking the 105 nginx configs"
fi
exit "${status}"
}
trap rollback ERR
@@ -323,9 +408,21 @@ shasum -a 256 -c "${PERSIST_BACKUP_TAR}.sha256"
gzip -t "${PERSIST_BACKUP_TAR}"
tar -tzf "${PERSIST_BACKUP_TAR}" | grep -E '^(\./)?\.env$' >/dev/null
if [[ -f "${TUNNEL_PLIST}" ]]; then
cp "${TUNNEL_PLIST}" "${TUNNEL_PLIST_BACKUP}"
fi
say "Create and verify the active 105 nginx routing backup"
edge_ssh \
"set -euo pipefail
test -f '${EDGE_MOBILE_CONFIG}'
test -f '${EDGE_WECHAT_CONFIG}'
grep -q 'proxy_pass http://58.38.22.103:8081;' '${EDGE_MOBILE_CONFIG}'
grep -q 'proxy_pass http://58.38.22.103:8081;' '${EDGE_WECHAT_CONFIG}'
! grep -q 'proxy_pass http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT};' '${EDGE_MOBILE_CONFIG}'
! grep -q 'proxy_pass http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT};' '${EDGE_WECHAT_CONFIG}'
cp '${EDGE_MOBILE_CONFIG}' '${EDGE_MOBILE_BACKUP}'
cp '${EDGE_WECHAT_CONFIG}' '${EDGE_WECHAT_BACKUP}'
sha256sum '${EDGE_MOBILE_BACKUP}' '${EDGE_WECHAT_BACKUP}' \
> '${EDGE_MOBILE_BACKUP}.sha256'
sha256sum -c '${EDGE_MOBILE_BACKUP}.sha256'
nginx -t >/dev/null"
say "Extract the source-free candidate runtime"
rm -rf "${CANDIDATE_DIR}"
@@ -334,13 +431,11 @@ tar -xzf "${BUNDLE}" -C "${CANDIDATE_DIR}"
cp "${MANIFEST}" "${CANDIDATE_DIR}/.release-manifest.txt"
chmod 755 \
"${CANDIDATE_DIR}/scripts/run-memind-portal-candidate.sh" \
"${CANDIDATE_DIR}/scripts/run-memind-canary-proxy-prod.sh"
"${CANDIDATE_DIR}/scripts/run-memind-canary-proxy-prod.sh" \
"${CANDIDATE_DIR}/scripts/run-deepseek-compat-proxy-candidate.sh"
say "Stop an older canary without touching stable Portal 8081"
if curl -fsS http://127.0.0.1:18080/api/status >/dev/null 2>&1; then
write_tunnel_plist 8081
sleep 2
fi
restore_edge_to_stable
stop_candidate_services
say "Start an isolated goosed candidate on 18015"
@@ -366,6 +461,39 @@ done
"${DOCKER_BIN}" exec goosed-prod-canary \
sh -lc 'test -x /usr/local/bin/node && test -f /opt/portal/mindspace-sandbox-mcp.mjs'
say "Start the DeepSeek tool-round compatibility proxy on 18036"
cat > "${DEEPSEEK_COMPAT_PLIST}" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key><string>${DEEPSEEK_COMPAT_LABEL}</string>
<key>ProgramArguments</key>
<array><string>${CANDIDATE_DIR}/scripts/run-deepseek-compat-proxy-candidate.sh</string></array>
<key>EnvironmentVariables</key>
<dict>
<key>MEMIND_CANARY_STABLE_ROOT</key><string>${STABLE_DIR}</string>
<key>MEMIND_CANARY_DEEPSEEK_PROXY_PORT</key><string>${DEEPSEEK_COMPAT_PORT}</string>
</dict>
<key>RunAtLoad</key><true/>
<key>KeepAlive</key><true/>
<key>ThrottleInterval</key><integer>5</integer>
<key>StandardOutPath</key><string>${HOME}/Library/Logs/memind-deepseek-compat-candidate.log</string>
<key>StandardErrorPath</key><string>${HOME}/Library/Logs/memind-deepseek-compat-candidate.log</string>
</dict>
</plist>
EOF
launchctl bootstrap "${LAUNCHD_GUI}" "${DEEPSEEK_COMPAT_PLIST}" >/dev/null
launchctl kickstart -k "${LAUNCHD_GUI}/${DEEPSEEK_COMPAT_LABEL}" >/dev/null
for _ in $(seq 1 30); do
deepseek_compat_healthy && break
sleep 1
done
deepseek_compat_healthy
"${DOCKER_BIN}" exec goosed-prod-canary \
sh -lc "curl -fsS --max-time 5 http://host.docker.internal:${DEEPSEEK_COMPAT_PORT}/health \
| grep -q '\"deepseekThinking\":\"disabled\"'"
say "Start the passive candidate Portal on 18081"
cat > "${CANDIDATE_PLIST}" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
@@ -381,6 +509,7 @@ cat > "${CANDIDATE_PLIST}" <<EOF
<key>MEMIND_CANARY_RELEASE_ID</key><string>${RELEASE_ID}</string>
<key>MEMIND_CANARY_CANDIDATE_PORT</key><string>18081</string>
<key>MEMIND_CANARY_GOOSED_URL</key><string>https://127.0.0.1:18015</string>
<key>MEMIND_CANARY_DEEPSEEK_PROXY_PORT</key><string>${DEEPSEEK_COMPAT_PORT}</string>
</dict>
<key>RunAtLoad</key><true/>
<key>KeepAlive</key><true/>
@@ -416,7 +545,7 @@ chmod 600 "${SECRET_FILE}"
/opt/homebrew/opt/node@24/bin/node "${CANDIDATE_DIR}/memind-canary-proxy.mjs" --check
)
say "Start the fail-closed identity router on 18080"
say "Start the fail-closed identity router on 18082"
cat > "${PROXY_PLIST}" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
@@ -429,8 +558,10 @@ cat > "${PROXY_PLIST}" <<EOF
<dict>
<key>MEMIND_CANARY_STABLE_ROOT</key><string>${STABLE_DIR}</string>
<key>MEMIND_CANARY_SECRET_FILE</key><string>${SECRET_FILE}</string>
<key>MEMIND_CANARY_PROXY_PORT</key><string>${CANARY_PROXY_PORT}</string>
<key>MEMIND_RELEASE_CANARY_USERNAMES</key><string>${CANARY_USERNAMES}</string>
<key>MEMIND_RELEASE_CANARY_WECHAT_USER_IDS</key><string>${CANARY_WECHAT_USER_IDS}</string>
<key>MEMIND_CANARY_CANDIDATE_HEALTH_URLS</key><string>http://127.0.0.1:${DEEPSEEK_COMPAT_PORT}/health</string>
</dict>
<key>RunAtLoad</key><true/>
<key>KeepAlive</key><true/>
@@ -447,7 +578,7 @@ diagnostic_secret="$(tr -d '\r\n' < "${SECRET_FILE}")"
diagnostic="$(
curl -fsS \
-H "X-Memind-Canary-Secret: ${diagnostic_secret}" \
http://127.0.0.1:18080/__memind_canary/health
"http://127.0.0.1:${CANARY_PROXY_PORT}/__memind_canary/health"
)"
printf '%s' "${diagnostic}" | /opt/homebrew/opt/node@24/bin/node --input-type=module -e '
let input = "";
@@ -462,7 +593,7 @@ for probe_query in 'username=john' 'wechat_user_id=wx_ul610et8'; do
route_probe="$(
curl -fsS \
-H "X-Memind-Canary-Secret: ${diagnostic_secret}" \
"http://127.0.0.1:18080/__memind_canary/route-probe?${probe_query}"
"http://127.0.0.1:${CANARY_PROXY_PORT}/__memind_canary/route-probe?${probe_query}"
)"
printf '%s' "${route_probe}" | /opt/homebrew/opt/node@24/bin/node --input-type=module -e '
let input = "";
@@ -475,7 +606,21 @@ for probe_query in 'username=john' 'wechat_user_id=wx_ul610et8'; do
'
done
say "Drain active Agent runs for the bounded tunnel switch"
say "Start the isolated 105 reverse tunnel on 19082"
write_canary_tunnel_plist
for _ in $(seq 1 30); do
edge_code="$(
edge_ssh \
"curl -sS -o /dev/null -w '%{http_code}' \
--max-time 5 http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT}/api/status" \
2>/dev/null || true
)"
[[ "${edge_code}" == "200" ]] && break
sleep 2
done
[[ "${edge_code:-}" == "200" ]]
say "Drain active Agent runs for the bounded edge switch"
touch "${STABLE_DIR}/.release-drain"
deadline=$(( $(date +%s) + 120 ))
while (( $(date +%s) < deadline )); do
@@ -504,15 +649,33 @@ while (( $(date +%s) < deadline )); do
done
[[ "${counts:-unknown unknown}" == "0 0" ]]
say "Switch only the reverse tunnel from stable 8081 to identity router 18080"
write_tunnel_plist 18080
sleep 2
remote_code="$(
ssh -o BatchMode=yes -o ConnectTimeout=10 "${MEMIND_PORTAL_TUNNEL_HOST}" \
"curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}/api/status" \
2>/dev/null || true
)"
[[ "${remote_code}" == "200" ]]
say "Switch the committed 105 nginx upstreams to the isolated canary tunnel"
# Unlike ordinary edge_ssh calls, this command intentionally receives only the
# bounded EDGE_SWITCH heredoc on stdin.
ssh -o BatchMode=yes -o ConnectTimeout=10 "${EDGE_HOST}" /bin/bash <<EDGE_SWITCH
set -euo pipefail
for config in '${EDGE_MOBILE_CONFIG}' '${EDGE_WECHAT_CONFIG}'; do
before_count="\$(grep -cF 'proxy_pass http://58.38.22.103:8081;' "\${config}")"
[[ "\${before_count}" -gt 0 ]]
temp_config="\$(mktemp "\${config}.canary.XXXXXX")"
sed 's#proxy_pass http://58\.38\.22\.103:8081;#proxy_pass http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT};#g' \
"\${config}" > "\${temp_config}"
after_count="\$(grep -cF 'proxy_pass http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT};' "\${temp_config}")"
[[ "\${after_count}" == "\${before_count}" ]]
chown root:root "\${temp_config}"
chmod 0644 "\${temp_config}"
mv "\${temp_config}" "\${config}"
done
nginx -t
systemctl reload nginx
curl -fsS --max-time 10 http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT}/api/status >/dev/null
curl -kfsS --max-time 15 \
--resolve m.tkmind.cn:443:127.0.0.1 \
https://m.tkmind.cn/api/status >/dev/null
curl -kfsS --max-time 15 \
--resolve wechat.m.tkmind.cn:443:127.0.0.1 \
https://wechat.m.tkmind.cn/api/status >/dev/null
EDGE_SWITCH
rm -f "${STABLE_DIR}/.release-drain"
say "Verify candidate failure automatically falls back to stable"
@@ -526,7 +689,7 @@ done
fallback_probe="$(
curl -fsS \
-H "X-Memind-Canary-Secret: ${diagnostic_secret}" \
'http://127.0.0.1:18080/__memind_canary/route-probe?username=john'
"http://127.0.0.1:${CANARY_PROXY_PORT}/__memind_canary/route-probe?username=john"
)"
printf '%s' "${fallback_probe}" | /opt/homebrew/opt/node@24/bin/node --input-type=module -e '
let input = "";
@@ -537,6 +700,9 @@ printf '%s' "${fallback_probe}" | /opt/homebrew/opt/node@24/bin/node --input-typ
if (status.candidate !== false || status.route !== "stable") process.exit(1);
});
'
edge_ssh \
"curl -fsS --max-time 10 \
http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT}/api/status >/dev/null"
launchctl bootstrap "${LAUNCHD_GUI}" "${CANDIDATE_PLIST}" >/dev/null
launchctl kickstart -k "${LAUNCHD_GUI}/${PORTAL_CANDIDATE_LABEL}" >/dev/null
for _ in $(seq 1 60); do
@@ -547,7 +713,7 @@ candidate_healthy
final_diagnostic="$(
curl -fsS \
-H "X-Memind-Canary-Secret: ${diagnostic_secret}" \
http://127.0.0.1:18080/__memind_canary/health
"http://127.0.0.1:${CANARY_PROXY_PORT}/__memind_canary/health"
)"
printf '%s' "${final_diagnostic}" | /opt/homebrew/opt/node@24/bin/node --input-type=module -e '
let input = "";
@@ -564,10 +730,14 @@ printf 'release_id=%s\n' "${RELEASE_ID}"
printf 'candidate_dir=%s\n' "${CANDIDATE_DIR}"
printf 'full_backup=%s\n' "${FULL_BACKUP_TAR}"
printf 'persist_backup=%s\n' "${PERSIST_BACKUP_TAR}"
printf 'edge_mobile_backup=%s:%s\n' "${EDGE_HOST}" "${EDGE_MOBILE_BACKUP}"
printf 'edge_wechat_backup=%s:%s\n' "${EDGE_HOST}" "${EDGE_WECHAT_BACKUP}"
printf 'stable_health=http://127.0.0.1:8081/api/status\n'
printf 'proxy_health=http://127.0.0.1:18080/__memind_canary/health\n'
printf 'proxy_health=http://127.0.0.1:%s/__memind_canary/health\n' "${CANARY_PROXY_PORT}"
printf 'edge_tunnel=http://127.0.0.1:%s/api/status\n' "${CANARY_TUNNEL_REMOTE_PORT}"
printf 'candidate_health=http://127.0.0.1:18081/api/status\n'
printf 'candidate_goosed=https://127.0.0.1:18015/status\n'
printf 'deepseek_compat_health=http://127.0.0.1:%s/health\n' "${DEEPSEEK_COMPAT_PORT}"
REMOTE_SCRIPT
say "103 canary release completed"
+1 -1
View File
@@ -166,7 +166,7 @@ fi
verify_runtime_artifact() {
local missing=0
for required in server.mjs memind-canary-proxy.mjs wechat-mp.bundle.mjs mindspace-sandbox-mcp.mjs tkmind-search-mcp.mjs tkmind-excel-mcp.mjs mindspace-public-links.mjs dist package.json scripts/run-memind-portal-prod.sh scripts/run-memind-portal-candidate.sh scripts/run-memind-canary-proxy-prod.sh scripts/goosed-canary.compose.yml scripts/check-mindspace-public-links.mjs scripts/load-env.mjs scripts/wechat-mp-menu.mjs scripts/memind-portal-tunnel.sh; do
for required in server.mjs memind-canary-proxy.mjs deepseek-no-think-proxy.mjs wechat-mp.bundle.mjs mindspace-sandbox-mcp.mjs tkmind-search-mcp.mjs tkmind-excel-mcp.mjs mindspace-public-links.mjs dist package.json scripts/run-memind-portal-prod.sh scripts/run-memind-portal-candidate.sh scripts/run-memind-canary-proxy-prod.sh scripts/run-deepseek-compat-proxy-candidate.sh scripts/goosed-canary.compose.yml scripts/check-mindspace-public-links.mjs scripts/load-env.mjs scripts/wechat-mp-menu.mjs scripts/memind-portal-tunnel.sh; do
if [[ ! -e "${RUNTIME_ROOT}/${required}" ]]; then
echo "runtime 产物缺失: ${RUNTIME_ROOT}/${required}" >&2
missing=1
+43 -41
View File
@@ -5,9 +5,12 @@ HOST="${STUDIO_HOST:-58.38.22.103}"
STABLE_DIR="${STUDIO_REMOTE_ROOT:-/Users/john/Project}/Memind"
PORTAL_CANDIDATE_LABEL="cn.tkmind.memind-portal-candidate"
CANARY_PROXY_LABEL="cn.tkmind.memind-canary-proxy"
PORTAL_TUNNEL_LABEL="cn.tkmind.memind-portal-tunnel"
MEMIND_PORTAL_TUNNEL_HOST="${MEMIND_PORTAL_TUNNEL_HOST:-ssh105-public}"
MEMIND_PORTAL_TUNNEL_REMOTE_PORT="${MEMIND_PORTAL_TUNNEL_REMOTE_PORT:-19081}"
CANARY_TUNNEL_LABEL="cn.tkmind.memind-canary-tunnel"
DEEPSEEK_COMPAT_LABEL="cn.tkmind.memind-deepseek-compat-candidate"
EDGE_HOST="${MEMIND_CANARY_EDGE_HOST:-ssh105-public}"
CANARY_TUNNEL_REMOTE_PORT=19082
EDGE_MOBILE_CONFIG="/etc/nginx/conf.d/m.tkmind.cn.conf"
EDGE_WECHAT_CONFIG="/etc/nginx/conf.d/wechat.m.tkmind.cn.conf"
AUTO_YES=0
if [[ "${1:-}" == "--yes" || "${1:-}" == "-y" ]]; then
@@ -18,7 +21,7 @@ elif [[ $# -gt 0 ]]; then
fi
if [[ "${AUTO_YES}" -ne 1 ]]; then
echo "This restores the 105 tunnel to stable Portal 8081 and stops only canary services."
echo "This restores both 105 nginx upstreams to stable Portal 8081 and stops only canary services."
read -r -p "Continue with 103 canary rollback? [y/N] " confirm </dev/tty
[[ "${confirm}" =~ ^[Yy]$ ]] || exit 0
fi
@@ -27,55 +30,54 @@ ssh -o BatchMode=yes -o ConnectTimeout=15 "${HOST}" \
"STABLE_DIR='${STABLE_DIR}' \
PORTAL_CANDIDATE_LABEL='${PORTAL_CANDIDATE_LABEL}' \
CANARY_PROXY_LABEL='${CANARY_PROXY_LABEL}' \
PORTAL_TUNNEL_LABEL='${PORTAL_TUNNEL_LABEL}' \
MEMIND_PORTAL_TUNNEL_HOST='${MEMIND_PORTAL_TUNNEL_HOST}' \
MEMIND_PORTAL_TUNNEL_REMOTE_PORT='${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}' \
CANARY_TUNNEL_LABEL='${CANARY_TUNNEL_LABEL}' \
DEEPSEEK_COMPAT_LABEL='${DEEPSEEK_COMPAT_LABEL}' \
EDGE_HOST='${EDGE_HOST}' \
CANARY_TUNNEL_REMOTE_PORT='${CANARY_TUNNEL_REMOTE_PORT}' \
EDGE_MOBILE_CONFIG='${EDGE_MOBILE_CONFIG}' \
EDGE_WECHAT_CONFIG='${EDGE_WECHAT_CONFIG}' \
/bin/bash" <<'REMOTE'
set -euo pipefail
LAUNCHD_GUI="gui/$(id -u)"
TUNNEL_PLIST="${HOME}/Library/LaunchAgents/${PORTAL_TUNNEL_LABEL}.plist"
ssh -o BatchMode=yes -o ConnectTimeout=10 "${EDGE_HOST}" /bin/bash <<EDGE_ROLLBACK
set -euo pipefail
for config in '${EDGE_MOBILE_CONFIG}' '${EDGE_WECHAT_CONFIG}'; do
canary_count="\$(grep -cF 'proxy_pass http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT};' "\${config}" || true)"
stable_count_before="\$(grep -cF 'proxy_pass http://58.38.22.103:8081;' "\${config}" || true)"
if [[ "\${canary_count}" -eq 0 ]]; then
[[ "\${stable_count_before}" -gt 0 ]]
continue
fi
temp_config="\$(mktemp "\${config}.stable.XXXXXX")"
sed 's#proxy_pass http://127\.0\.0\.1:${CANARY_TUNNEL_REMOTE_PORT};#proxy_pass http://58.38.22.103:8081;#g' \
"\${config}" > "\${temp_config}"
stable_count="\$(grep -cF 'proxy_pass http://58.38.22.103:8081;' "\${temp_config}")"
[[ "\${stable_count}" -eq \$(( stable_count_before + canary_count )) ]]
chown root:root "\${temp_config}"
chmod 0644 "\${temp_config}"
mv "\${temp_config}" "\${config}"
done
nginx -t
systemctl reload nginx
curl -kfsS --max-time 15 \
--resolve m.tkmind.cn:443:127.0.0.1 \
https://m.tkmind.cn/api/status >/dev/null
curl -kfsS --max-time 15 \
--resolve wechat.m.tkmind.cn:443:127.0.0.1 \
https://wechat.m.tkmind.cn/api/status >/dev/null
EDGE_ROLLBACK
cat > "${TUNNEL_PLIST}" <<EOF
<?xml version="1.0" encoding="UTF-8"?>
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
<plist version="1.0">
<dict>
<key>Label</key><string>${PORTAL_TUNNEL_LABEL}</string>
<key>ProgramArguments</key>
<array><string>${STABLE_DIR}/scripts/memind-portal-tunnel.sh</string></array>
<key>EnvironmentVariables</key>
<dict>
<key>MEMIND_PORTAL_TUNNEL_HOST</key><string>${MEMIND_PORTAL_TUNNEL_HOST}</string>
<key>MEMIND_PORTAL_TUNNEL_LOCAL_PORT</key><string>8081</string>
<key>MEMIND_PORTAL_TUNNEL_REMOTE_PORT</key><string>${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}</string>
</dict>
<key>RunAtLoad</key><true/>
<key>KeepAlive</key><true/>
<key>StandardOutPath</key><string>${HOME}/Library/Logs/memind-portal-tunnel.log</string>
<key>StandardErrorPath</key><string>${HOME}/Library/Logs/memind-portal-tunnel.log</string>
</dict>
</plist>
EOF
launchctl bootout "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null 2>&1 || true
launchctl bootstrap "${LAUNCHD_GUI}" "${TUNNEL_PLIST}" >/dev/null
launchctl kickstart -k "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null
launchctl bootout "${LAUNCHD_GUI}/${CANARY_TUNNEL_LABEL}" >/dev/null 2>&1 || true
launchctl bootout "${LAUNCHD_GUI}/${CANARY_PROXY_LABEL}" >/dev/null 2>&1 || true
launchctl bootout "${LAUNCHD_GUI}/${PORTAL_CANDIDATE_LABEL}" >/dev/null 2>&1 || true
launchctl bootout "${LAUNCHD_GUI}/${DEEPSEEK_COMPAT_LABEL}" >/dev/null 2>&1 || true
/opt/homebrew/bin/docker rm -f goosed-prod-canary >/dev/null 2>&1 || true
rm -f "${STABLE_DIR}/.release-drain"
curl -fsS http://127.0.0.1:8081/api/status >/dev/null
sleep 2
remote_code="$(
ssh -o BatchMode=yes -o ConnectTimeout=10 "${MEMIND_PORTAL_TUNNEL_HOST}" \
"curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}/api/status" \
2>/dev/null || true
)"
[[ "${remote_code}" == "200" ]]
printf 'stable_port=8081\n'
printf 'remote_tunnel_status=%s\n' "${remote_code}"
printf 'edge_upstream=58.38.22.103:8081\n'
REMOTE
echo "103 canary rollback completed; stable Portal remains active on 8081."
+28
View File
@@ -0,0 +1,28 @@
#!/usr/bin/env bash
set -euo pipefail
ROOT="$(cd "$(dirname "$0")/.." && pwd)"
STABLE_ROOT="${MEMIND_CANARY_STABLE_ROOT:-/Users/john/Project/Memind}"
if [[ ! -d "${STABLE_ROOT}" || ! -f "${STABLE_ROOT}/.env" ]]; then
echo "[deepseek-compat] stable runtime environment is unavailable: ${STABLE_ROOT}" >&2
exit 1
fi
cd "${ROOT}"
set -a
# shellcheck disable=SC1091
source "${STABLE_ROOT}/.env"
set +a
export NODE_ENV=production
export MEMIND_DEEPSEEK_PROXY_ENTRYPOINT=1
export MEMIND_DEEPSEEK_NO_THINK_PORT="${MEMIND_CANARY_DEEPSEEK_PROXY_PORT:-18036}"
NODE_BIN="${NODE_BIN:-/opt/homebrew/opt/node@24/bin/node}"
if [[ ! -x "${NODE_BIN}" ]]; then
NODE_BIN="$(command -v node)"
fi
exec "${NODE_BIN}" "${ROOT}/deepseek-no-think-proxy.mjs"
+1 -1
View File
@@ -23,7 +23,7 @@ export NODE_ENV=production
export MEMIND_PORTAL_H5_ROOT="${STABLE_ROOT}"
export MEMIND_CANARY_DIAGNOSTIC_SECRET="$(tr -d '\r\n' < "${SECRET_FILE}")"
export MEMIND_CANARY_PROXY_HOST=127.0.0.1
export MEMIND_CANARY_PROXY_PORT="${MEMIND_CANARY_PROXY_PORT:-18080}"
export MEMIND_CANARY_PROXY_PORT="${MEMIND_CANARY_PROXY_PORT:-18082}"
export MEMIND_CANARY_STABLE_URL="${MEMIND_CANARY_STABLE_URL:-http://127.0.0.1:8081}"
export MEMIND_CANARY_CANDIDATE_URL="${MEMIND_CANARY_CANDIDATE_URL:-http://127.0.0.1:18081}"
+13
View File
@@ -9,6 +9,8 @@ if [[ ! -d "${STABLE_ROOT}" || ! -f "${STABLE_ROOT}/.env" ]]; then
exit 1
fi
cd "${ROOT}"
set -a
# shellcheck disable=SC1091
source "${STABLE_ROOT}/.env"
@@ -27,6 +29,17 @@ export H5_HOST=127.0.0.1
export H5_PUBLIC_BASE_URL="${H5_PUBLIC_BASE_URL:-https://m.tkmind.cn}"
export TKMIND_API_TARGETS="${MEMIND_CANARY_GOOSED_URL:-https://127.0.0.1:18015}"
export TKMIND_API_TARGET="${MEMIND_CANARY_GOOSED_URL:-https://127.0.0.1:18015}"
# Extensions are spawned inside goosed-canary, where the candidate artifact is
# mounted at /opt/portal. Never inherit host-only MCP paths from the stable .env.
export GOOSED_MCP_NODE_PATH=/usr/local/bin/node
export GOOSED_MCP_SERVER_PATH=/opt/portal/mindspace-sandbox-mcp.mjs
# DeepSeek V4 tool rounds must use the same compatibility contract exercised
# by the release Gate. Stable .env values cannot disable or redirect it.
export MEMIND_DEEPSEEK_DISABLE_THINKING=1
export MEMIND_DEEPSEEK_NO_THINK_PORT="${MEMIND_CANARY_DEEPSEEK_PROXY_PORT:-18036}"
export MEMIND_GOOSED_HOST_GATEWAY=host.docker.internal
unset MEMIND_DEEPSEEK_NO_THINK_BASE_URL
unset MEMIND_DEEPSEEK_NO_THINK_HOST
unset TKMIND_API_TARGET_1
NODE_BIN="${NODE_BIN:-/opt/homebrew/opt/node@24/bin/node}"