Compare commits
2 Commits
| Author | SHA1 | Date | |
|---|---|---|---|
| 02f2dc6b17 | |||
| 473ad80882 |
@@ -1,6 +1,6 @@
|
||||
# Local and 103 runtime topology
|
||||
|
||||
> Last confirmed: 2026-07-07 09:19 CST.
|
||||
> Last confirmed: 2026-07-26 20:18 CST.
|
||||
>
|
||||
> This is the current topology source of truth for local Memind and 103 production. Prefer this document over older migration notes. Older architecture documents may contain historical paths from before the MindSpace split.
|
||||
|
||||
@@ -39,14 +39,20 @@ Portal must be released as a runtime artifact. Do not edit source directly on 10
|
||||
|
||||
| Item | Value |
|
||||
|------|-------|
|
||||
| Identity router | `127.0.0.1:18080` |
|
||||
| Stable edge upstream | `105 nginx -> 58.38.22.103:8081` |
|
||||
| Identity router | `127.0.0.1:18082` |
|
||||
| Canary edge tunnel | `105 127.0.0.1:19082 -> 103 127.0.0.1:18082` |
|
||||
| Candidate Portal | `127.0.0.1:18081` |
|
||||
| Candidate goosed | `127.0.0.1:18015` |
|
||||
| Release entry | `scripts/release-portal-canary-prod.sh` |
|
||||
| Rollback entry | `scripts/rollback-portal-canary-prod.sh` |
|
||||
|
||||
The stable Portal stays on 8081. The 105 reverse tunnel points to 18080 only after the
|
||||
candidate, router, backups, immutable identity checks, and fallback checks pass.
|
||||
The stable Portal stays on 8081 and the normal 105 edge path is a direct upstream, not
|
||||
the retired 19081 tunnel. A canary release starts an isolated 19082 tunnel and changes
|
||||
the two committed 105 nginx upstreams only after the candidate, router, 103/105 backups,
|
||||
immutable identity checks, and tunnel checks pass. It then proves stable fallback before
|
||||
declaring success. Port 18080 is occupied by a separate Memory V2 mock service and must
|
||||
not be reused.
|
||||
|
||||
## MindSpace Service
|
||||
|
||||
|
||||
@@ -114,6 +114,22 @@ ssh root@120.26.184.105 '
|
||||
- 禁止在 `105` 直接改 `server.mjs` / `wechat-mp.mjs` / `scripts/wechat-mp-menu.mjs`
|
||||
- 禁止跳过健康检查就宣称“已发布”
|
||||
|
||||
## 103 Portal 用户灰度的 105 入口切换
|
||||
|
||||
103 Portal 灰度不发布业务源码到 105。正式入口由已提交的
|
||||
`scripts/release-portal-canary-prod.sh` 受控变更:
|
||||
|
||||
1. 先备份并校验 105 活动的 `m.tkmind.cn.conf` 与 `wechat.m.tkmind.cn.conf`。
|
||||
2. 在 103 启动独立路由器 `18082`,并通过反向隧道只暴露为 105 本机 `19082`。
|
||||
3. 候选、身份路由和隧道全部通过后,脚本才把两份 nginx 上游从
|
||||
`58.38.22.103:8081` 切到 `127.0.0.1:19082`。
|
||||
4. 必须先 `nginx -t`,再 reload,并在切换后主动证明候选故障会回落稳定 8081;
|
||||
任一步失败恢复备份并回到稳定入口。
|
||||
5. 回滚只能使用 `scripts/rollback-portal-canary-prod.sh`,禁止在 105 手工 `sed -i`。
|
||||
|
||||
该动作属于 commit、CI、完整 Gate report 和明确生产批准约束下的正式发布运维,
|
||||
不构成允许在线编辑 105 配置源码的一般例外。
|
||||
|
||||
## 服务号底部菜单(`wechat-mp-menu.mjs`)
|
||||
|
||||
菜单名称与链接定义在本地:
|
||||
|
||||
@@ -545,7 +545,9 @@ failed = skipped = blocked = unknown = cleanup_failed = 0
|
||||
### 10.0 用户级灰度前置条件
|
||||
|
||||
用户级灰度入口由 `scripts/release-portal-canary-prod.sh` 安装:稳定 Portal 保持 8081,
|
||||
候选 Portal 使用 18081,候选 goosed 使用 18015,身份路由器使用 18080。灰度身份必须
|
||||
候选 Portal 使用 18081,候选 goosed 使用 18015,身份路由器使用 18082;105 使用独立
|
||||
19082 反向隧道访问路由器。活动 nginx 上游在脚本备份、`nginx -t` 和健康检查后,才允许
|
||||
从稳定 `58.38.22.103:8081` 切到 `127.0.0.1:19082`。灰度身份必须
|
||||
使用不可变用户 ID、唯一用户名或微信绑定账号标识;禁止按显示昵称(包括“唐”)匹配。未命中、
|
||||
身份解析失败或候选不可用必须自动回落稳定版本。
|
||||
|
||||
|
||||
+13
-10
@@ -6,8 +6,9 @@ Portal 的首个生产动作必须是用户级灰度,不能直接整包替换
|
||||
|
||||
| 组件 | 地址 | 说明 |
|
||||
|---|---|---|
|
||||
| 稳定 Portal | `127.0.0.1:8081` | 保持当前 live 目录和进程,不在灰度发布中替换 |
|
||||
| 灰度路由器 | `127.0.0.1:18080` | 105 的 `19081` 反向隧道在全部检查通过后切到这里 |
|
||||
| 稳定 Portal | `103:8081` | 105 当前直接回源 `58.38.22.103:8081`;灰度发布不替换该进程 |
|
||||
| 灰度路由器 | `103 127.0.0.1:18082` | 避开已占用的 18080;只通过专用灰度隧道暴露给 105 |
|
||||
| 灰度边缘隧道 | `105 127.0.0.1:19082` | 反向 SSH 到 103 的 18082;不复用已退役的 19081 链路 |
|
||||
| 候选 Portal | `127.0.0.1:18081` | 独立、无源码 runtime;使用被动运行时模式 |
|
||||
| 候选 goosed | `127.0.0.1:18015` | 独立容器并挂载候选 runtime,不重建稳定 9 个容器 |
|
||||
|
||||
@@ -40,14 +41,16 @@ bash scripts/release-portal-canary-prod.sh --yes
|
||||
1. 验证源码、分支、远端主线、Gate report 和 artifact hash。
|
||||
2. 执行 103 只读预检。
|
||||
3. 上传经过验证的候选包。
|
||||
4. 创建稳定 live 全量备份和持久目录备份,并校验 SHA256、gzip 和关键目录。
|
||||
4. 创建稳定 live 全量备份、持久目录备份和 105 两份活动 nginx 配置备份,并校验完整性。
|
||||
5. 启动独立候选 goosed、候选 Portal 和灰度路由器。
|
||||
6. 验证不可变身份配置、稳定/候选健康和路由器诊断。
|
||||
7. 短暂阻止新 Agent Run,排空活动任务。
|
||||
8. 只把反向隧道从 8081 切到 18080。
|
||||
9. 主动停止候选,证明目标身份会自动回落稳定版,再恢复候选。
|
||||
7. 启动独立的 `105:19082 -> 103:18082` 灰度隧道并从 105 验证。
|
||||
8. 短暂阻止新 Agent Run,排空活动任务。
|
||||
9. 逐文件原子替换、验证并 reload 105 nginx,把活动上游从 `58.38.22.103:8081` 切到 `127.0.0.1:19082`。
|
||||
10. 主动停止候选,证明目标身份会自动回落稳定版,再恢复候选。
|
||||
|
||||
任一步失败都会把隧道恢复到稳定 8081,停止候选服务并保留候选目录和备份证据。
|
||||
任一步失败都会把 105 nginx 恢复到稳定 `58.38.22.103:8081`,停止灰度隧道和候选服务,
|
||||
并保留候选目录与备份证据。活动 nginx 配置变更只能由已提交的发布脚本执行,禁止在线手改。
|
||||
|
||||
## 回滚
|
||||
|
||||
@@ -57,10 +60,10 @@ bash scripts/rollback-portal-canary-prod.sh --yes
|
||||
|
||||
回滚只执行以下动作:
|
||||
|
||||
- 把 105 反向隧道恢复到稳定 Portal 8081。
|
||||
- 停止灰度路由器、候选 Portal 和候选 goosed。
|
||||
- 把 105 两份活动 nginx 上游恢复到稳定 Portal `58.38.22.103:8081` 并验证 reload。
|
||||
- 停止灰度隧道、灰度路由器、候选 Portal 和候选 goosed。
|
||||
- 清除发布排水标记。
|
||||
- 验证稳定 Portal 和 105 隧道健康。
|
||||
- 验证稳定 Portal 和 105 HTTPS 入口健康。
|
||||
|
||||
不会删除候选目录、备份或稳定运行目录。
|
||||
|
||||
|
||||
+7
-4
@@ -61,17 +61,20 @@ bash scripts/release-portal-canary-prod.sh --yes
|
||||
- 上传 artifact 到 103。
|
||||
- 103 备份当前 `/Users/john/Project/Memind` 全目录。
|
||||
- 103 单独备份持久目录:`.env`、`MindSpace/`、`data/`、`users/`、`.tailscale/`、`public/plaza-covers/`、`logs/`。
|
||||
- 原子切换 live 目录。
|
||||
- 更新 LaunchAgent 指向 `/Users/john/Project/Memind/scripts/run-memind-portal-prod.sh`。
|
||||
- 健康检查 `http://127.0.0.1:8081/api/status` 返回 200。
|
||||
- 备份并校验 105 的 `m.tkmind.cn.conf` 与 `wechat.m.tkmind.cn.conf`。
|
||||
- 保持稳定 Portal `103:8081` 不变,启动候选 Portal `18081`、候选 goosed `18015` 和身份路由器 `18082`。
|
||||
- 启动独立 `105:19082 -> 103:18082` 灰度隧道;禁止复用历史 19081 或占用 18080。
|
||||
- `nginx -t` 通过后才把 105 活动上游从 `58.38.22.103:8081` 切到 `127.0.0.1:19082`。
|
||||
- 健康检查稳定、候选、回落和 105 HTTPS 入口均通过。
|
||||
- 校验 `MindSpace/*/public/*.html` 下载/附件相对链接均已落盘(`check-mindspace-public-links.mjs --downloads-only`)。
|
||||
|
||||
发布成功后,记录脚本输出里的:
|
||||
|
||||
- `release_id`
|
||||
- `archived_source`
|
||||
- `candidate_dir`
|
||||
- `full_backup`
|
||||
- `persist_backup`
|
||||
- 105 nginx 配置备份
|
||||
- `git_head`
|
||||
|
||||
## 3. Portal 发布后必须验收
|
||||
|
||||
@@ -534,7 +534,7 @@ async function main() {
|
||||
|| '',
|
||||
});
|
||||
const listenHost = process.env.MEMIND_CANARY_PROXY_HOST || '127.0.0.1';
|
||||
const listenPort = Number(process.env.MEMIND_CANARY_PROXY_PORT ?? 18080);
|
||||
const listenPort = Number(process.env.MEMIND_CANARY_PROXY_PORT ?? 18082);
|
||||
const proxy = createMemindCanaryProxy({
|
||||
stableTarget: process.env.MEMIND_CANARY_STABLE_URL || 'http://127.0.0.1:8081',
|
||||
candidateTarget:
|
||||
|
||||
@@ -6,6 +6,25 @@ import test from 'node:test';
|
||||
|
||||
const ROOT = path.resolve(new URL('..', import.meta.url).pathname);
|
||||
const CANARY_RELEASE = path.join(ROOT, 'scripts', 'release-portal-canary-prod.sh');
|
||||
const CANARY_ROLLBACK = path.join(ROOT, 'scripts', 'rollback-portal-canary-prod.sh');
|
||||
|
||||
function heredocBody(source, opener, closer) {
|
||||
const start = source.indexOf(opener);
|
||||
assert.ok(start >= 0, `missing heredoc opener: ${opener}`);
|
||||
const bodyStart = source.indexOf('\n', start) + 1;
|
||||
const end = source.indexOf(`\n${closer}\n`, bodyStart);
|
||||
assert.ok(bodyStart > 0 && end >= bodyStart, `missing heredoc closer: ${closer}`);
|
||||
return source.slice(bodyStart, end);
|
||||
}
|
||||
|
||||
function assertShellParses(source, label) {
|
||||
const result = spawnSync('bash', ['-n'], {
|
||||
cwd: ROOT,
|
||||
encoding: 'utf8',
|
||||
input: source,
|
||||
});
|
||||
assert.equal(result.status, 0, `${label} failed bash -n:\n${result.stderr}`);
|
||||
}
|
||||
|
||||
test('production release verifies gate report before any 103 connection', async () => {
|
||||
const source = await fs.readFile(
|
||||
@@ -96,28 +115,70 @@ test('production canary keeps stable 8081 live and switches only after verified
|
||||
const source = await fs.readFile(CANARY_RELEASE, 'utf8');
|
||||
const fullBackup = source.indexOf('Create and verify the full stable backup');
|
||||
const persistBackup = source.indexOf('Create and verify the persisted-data backup');
|
||||
const edgeBackup = source.indexOf('Create and verify the active 105 nginx routing backup');
|
||||
const goosedStart = source.indexOf('Start an isolated goosed candidate on 18015');
|
||||
const candidateStart = source.indexOf('Start the passive candidate Portal on 18081');
|
||||
const proxyStart = source.indexOf('Start the fail-closed identity router on 18080');
|
||||
const tunnelSwitch = source.indexOf('Switch only the reverse tunnel');
|
||||
const proxyStart = source.indexOf('Start the fail-closed identity router on 18082');
|
||||
const tunnelStart = source.indexOf('Start the isolated 105 reverse tunnel on 19082');
|
||||
const edgeSwitch = source.indexOf(
|
||||
'Switch the committed 105 nginx upstreams to the isolated canary tunnel',
|
||||
);
|
||||
const fallbackProbe = source.indexOf('route-probe?username=john');
|
||||
assert.ok(fullBackup > 0);
|
||||
assert.ok(persistBackup > fullBackup);
|
||||
assert.ok(goosedStart > persistBackup);
|
||||
assert.ok(edgeBackup > persistBackup);
|
||||
assert.ok(goosedStart > edgeBackup);
|
||||
assert.ok(candidateStart > goosedStart);
|
||||
assert.ok(proxyStart > candidateStart);
|
||||
assert.ok(tunnelSwitch > proxyStart);
|
||||
assert.ok(fallbackProbe > tunnelSwitch);
|
||||
assert.match(source, /write_tunnel_plist 8081/);
|
||||
assert.ok(tunnelStart > proxyStart);
|
||||
assert.ok(edgeSwitch > tunnelStart);
|
||||
assert.ok(fallbackProbe > edgeSwitch);
|
||||
assert.match(source, /restore_edge_to_stable/);
|
||||
assert.match(source, /proxy_pass http:\/\/58\.38\.22\.103:8081;/);
|
||||
assert.match(source, /proxy_pass http:\/\/127\.0\.0\.1:\$\{CANARY_TUNNEL_REMOTE_PORT\};/);
|
||||
assert.match(source, /nginx -t/);
|
||||
assert.match(source, /CANARY_PROXY_PORT=18082/);
|
||||
assert.match(source, /CANARY_TUNNEL_REMOTE_PORT=19082/);
|
||||
assert.match(
|
||||
source,
|
||||
/edge_ssh\(\) \{\s*#.*\n\s*#.*\n\s*ssh -n -o BatchMode=yes/,
|
||||
);
|
||||
assert.match(
|
||||
source,
|
||||
/ssh -o BatchMode=yes -o ConnectTimeout=10 "\$\{EDGE_HOST\}" \/bin\/bash <<EDGE_SWITCH/,
|
||||
);
|
||||
assert.doesNotMatch(source, /edge_ssh \/bin\/bash <<EDGE_SWITCH/);
|
||||
assert.match(source, /CANARY_USERNAMES="john"/);
|
||||
assert.match(source, /CANARY_WECHAT_USER_IDS="wx_ul610et8"/);
|
||||
assert.match(source, /route-probe\?\$\{probe_query\}/);
|
||||
assert.match(source, /candidate_healthy/);
|
||||
assert.match(source, /MEMIND_CANARY_RELEASE_ID/);
|
||||
assert.match(
|
||||
source,
|
||||
/if restore_edge_to_stable >\/dev\/null 2>&1; then\s+stop_candidate_services/,
|
||||
);
|
||||
assert.match(source, /keeping canary services running/);
|
||||
assert.match(source, /--resolve wechat\.m\.tkmind\.cn:443:127\.0\.0\.1/);
|
||||
assert.doesNotMatch(source, /lsof -tiTCP:8081/);
|
||||
assert.doesNotMatch(source, /bootout.*cn\.tkmind\.memind-portal/);
|
||||
});
|
||||
|
||||
test('canary release and rollback remote shells remain syntactically valid', async () => {
|
||||
const releaseSource = await fs.readFile(CANARY_RELEASE, 'utf8');
|
||||
const rollbackSource = await fs.readFile(CANARY_ROLLBACK, 'utf8');
|
||||
const releaseRemote = heredocBody(releaseSource, "<<'REMOTE_SCRIPT'", 'REMOTE_SCRIPT');
|
||||
const rollbackRemote = heredocBody(rollbackSource, "<<'REMOTE'", 'REMOTE');
|
||||
const edgeSwitch = heredocBody(releaseRemote, '<<EDGE_SWITCH', 'EDGE_SWITCH')
|
||||
.replaceAll('\\$', '$');
|
||||
const edgeRollback = heredocBody(rollbackRemote, '<<EDGE_ROLLBACK', 'EDGE_ROLLBACK')
|
||||
.replaceAll('\\$', '$');
|
||||
|
||||
assertShellParses(releaseRemote, 'release remote shell');
|
||||
assertShellParses(rollbackRemote, 'rollback remote shell');
|
||||
assertShellParses(edgeSwitch, '105 edge switch shell');
|
||||
assertShellParses(edgeRollback, '105 edge rollback shell');
|
||||
});
|
||||
|
||||
test('release readiness ignores only the generated canary artifact cover path', async () => {
|
||||
const source = await fs.readFile(
|
||||
path.join(ROOT, 'scripts', 'check-release-ready.sh'),
|
||||
|
||||
@@ -12,9 +12,12 @@ GOOSED_DIR="${REMOTE_ROOT}/goosed-prod"
|
||||
RUNTIME_ROOT="${ROOT}/.runtime/portal"
|
||||
PORTAL_CANDIDATE_LABEL="cn.tkmind.memind-portal-candidate"
|
||||
CANARY_PROXY_LABEL="cn.tkmind.memind-canary-proxy"
|
||||
PORTAL_TUNNEL_LABEL="cn.tkmind.memind-portal-tunnel"
|
||||
MEMIND_PORTAL_TUNNEL_HOST="${MEMIND_PORTAL_TUNNEL_HOST:-ssh105-public}"
|
||||
MEMIND_PORTAL_TUNNEL_REMOTE_PORT="${MEMIND_PORTAL_TUNNEL_REMOTE_PORT:-19081}"
|
||||
CANARY_TUNNEL_LABEL="cn.tkmind.memind-canary-tunnel"
|
||||
EDGE_HOST="${MEMIND_CANARY_EDGE_HOST:-ssh105-public}"
|
||||
CANARY_PROXY_PORT=18082
|
||||
CANARY_TUNNEL_REMOTE_PORT=19082
|
||||
EDGE_MOBILE_CONFIG="/etc/nginx/conf.d/m.tkmind.cn.conf"
|
||||
EDGE_WECHAT_CONFIG="/etc/nginx/conf.d/wechat.m.tkmind.cn.conf"
|
||||
CANARY_USERNAMES="john"
|
||||
CANARY_WECHAT_USER_IDS="wx_ul610et8"
|
||||
RELEASE_TS="$(date +%Y%m%d-%H%M%S)"
|
||||
@@ -40,13 +43,14 @@ Usage:
|
||||
|
||||
Deploys a candidate Portal beside the stable 103 runtime:
|
||||
stable Portal 127.0.0.1:8081
|
||||
canary router 127.0.0.1:18080
|
||||
canary router 127.0.0.1:18082
|
||||
candidate Portal 127.0.0.1:18081
|
||||
candidate goosed 127.0.0.1:18015
|
||||
|
||||
The stable runtime is not replaced. The 105 reverse tunnel moves to the canary
|
||||
router only after backups, candidate health, immutable identity checks, and
|
||||
automatic fallback checks pass.
|
||||
The stable runtime is not replaced. A dedicated reverse tunnel exposes the
|
||||
router only to 105 at 127.0.0.1:19082. The committed release workflow updates
|
||||
the two active 105 nginx upstreams only after backups, candidate health, and
|
||||
immutable identity checks pass, then proves automatic fallback before success.
|
||||
EOF
|
||||
}
|
||||
|
||||
@@ -151,7 +155,7 @@ printf '%s %s\n' "${bundle_sha}" "$(basename "${BUNDLE_PATH}")" > "${SHA_PATH}"
|
||||
echo "git_branch=${branch}"
|
||||
echo "artifact_tree=.runtime/portal"
|
||||
echo "artifact_bundle_sha256=${bundle_sha}"
|
||||
echo "routing=stable:8081,proxy:18080,candidate:18081,goosed-canary:18015"
|
||||
echo "routing=stable:8081,proxy:18082,edge-tunnel:19082,candidate:18081,goosed-canary:18015"
|
||||
echo "canary_usernames=${CANARY_USERNAMES}"
|
||||
echo "canary_wechat_user_ids=${CANARY_WECHAT_USER_IDS}"
|
||||
echo "manual_environment_changes=none"
|
||||
@@ -169,15 +173,28 @@ ssh -o BatchMode=yes -o ConnectTimeout=15 "${HOST}" \
|
||||
&& test -f '${STABLE_DIR}/.env' \
|
||||
&& test -f '${GOOSED_DIR}/docker-compose.prod.yml' \
|
||||
&& curl -fsS http://127.0.0.1:8081/api/status >/dev/null \
|
||||
&& ! lsof -nP -iTCP:${CANARY_PROXY_PORT} -sTCP:LISTEN >/dev/null 2>&1 \
|
||||
&& ! lsof -nP -iTCP:18081 -sTCP:LISTEN >/dev/null 2>&1 \
|
||||
&& ! lsof -nP -iTCP:18015 -sTCP:LISTEN >/dev/null 2>&1 \
|
||||
&& /opt/homebrew/bin/docker inspect goosed-prod-1 >/dev/null \
|
||||
&& test \"\$(df -Pk '${REMOTE_ROOT}' | awk 'NR==2 {print \$4}')\" -gt 10485760"
|
||||
&& test \"\$(df -Pk '${REMOTE_ROOT}' | awk 'NR==2 {print \$4}')\" -gt 10485760 \
|
||||
&& ssh -o BatchMode=yes -o ConnectTimeout=10 '${EDGE_HOST}' \
|
||||
\"test -f '${EDGE_MOBILE_CONFIG}' \
|
||||
&& test -f '${EDGE_WECHAT_CONFIG}' \
|
||||
&& grep -q 'proxy_pass http://58.38.22.103:8081;' '${EDGE_MOBILE_CONFIG}' \
|
||||
&& grep -q 'proxy_pass http://58.38.22.103:8081;' '${EDGE_WECHAT_CONFIG}' \
|
||||
&& ! grep -q 'proxy_pass http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT};' '${EDGE_MOBILE_CONFIG}' \
|
||||
&& ! grep -q 'proxy_pass http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT};' '${EDGE_WECHAT_CONFIG}' \
|
||||
&& nginx -t >/dev/null 2>&1 \
|
||||
&& ! lsof -nP -iTCP:${CANARY_TUNNEL_REMOTE_PORT} -sTCP:LISTEN >/dev/null 2>&1\""
|
||||
|
||||
if [[ "${AUTO_YES}" -ne 1 ]]; then
|
||||
say "Production canary confirmation"
|
||||
echo "Target host: ${HOST}"
|
||||
echo "Release: ${RELEASE_ID}"
|
||||
echo "Commit: ${FULL_SHA}"
|
||||
echo "Stable remains on 8081; selected immutable identities route to candidate 18081."
|
||||
echo "Stable remains on 8081; 105 will use the isolated 19082 canary tunnel."
|
||||
echo "Selected immutable identities route to candidate 18081."
|
||||
read -r -p "Continue with the 103 canary release? [y/N] " confirm </dev/tty
|
||||
[[ "${confirm}" =~ ^[Yy]$ ]] || exit 0
|
||||
fi
|
||||
@@ -196,9 +213,12 @@ ssh -o BatchMode=yes "${HOST}" \
|
||||
GOOSED_DIR='${GOOSED_DIR}' \
|
||||
PORTAL_CANDIDATE_LABEL='${PORTAL_CANDIDATE_LABEL}' \
|
||||
CANARY_PROXY_LABEL='${CANARY_PROXY_LABEL}' \
|
||||
PORTAL_TUNNEL_LABEL='${PORTAL_TUNNEL_LABEL}' \
|
||||
MEMIND_PORTAL_TUNNEL_HOST='${MEMIND_PORTAL_TUNNEL_HOST}' \
|
||||
MEMIND_PORTAL_TUNNEL_REMOTE_PORT='${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}' \
|
||||
CANARY_TUNNEL_LABEL='${CANARY_TUNNEL_LABEL}' \
|
||||
EDGE_HOST='${EDGE_HOST}' \
|
||||
CANARY_PROXY_PORT='${CANARY_PROXY_PORT}' \
|
||||
CANARY_TUNNEL_REMOTE_PORT='${CANARY_TUNNEL_REMOTE_PORT}' \
|
||||
EDGE_MOBILE_CONFIG='${EDGE_MOBILE_CONFIG}' \
|
||||
EDGE_WECHAT_CONFIG='${EDGE_WECHAT_CONFIG}' \
|
||||
CANARY_USERNAMES='${CANARY_USERNAMES}' \
|
||||
CANARY_WECHAT_USER_IDS='${CANARY_WECHAT_USER_IDS}' \
|
||||
/bin/bash" <<'REMOTE_SCRIPT'
|
||||
@@ -211,13 +231,14 @@ MANIFEST="${INCOMING_DIR}/memind-portal-canary-${RELEASE_ID}.manifest.txt"
|
||||
SHA_FILE="${INCOMING_DIR}/memind-portal-canary-${RELEASE_ID}.sha256"
|
||||
FULL_BACKUP_TAR="${BACKUP_DIR}/memind-full-${RELEASE_ID}-before-canary.tar.gz"
|
||||
PERSIST_BACKUP_TAR="${BACKUP_DIR}/memind-persisted-${RELEASE_ID}-before-canary.tar.gz"
|
||||
TUNNEL_PLIST="${HOME}/Library/LaunchAgents/${PORTAL_TUNNEL_LABEL}.plist"
|
||||
TUNNEL_PLIST_BACKUP="${BACKUP_DIR}/${PORTAL_TUNNEL_LABEL}-${RELEASE_ID}.plist"
|
||||
CANDIDATE_PLIST="${HOME}/Library/LaunchAgents/${PORTAL_CANDIDATE_LABEL}.plist"
|
||||
PROXY_PLIST="${HOME}/Library/LaunchAgents/${CANARY_PROXY_LABEL}.plist"
|
||||
CANARY_TUNNEL_PLIST="${HOME}/Library/LaunchAgents/${CANARY_TUNNEL_LABEL}.plist"
|
||||
SECRET_FILE="${HOME}/.config/memind/canary-router.secret"
|
||||
DOCKER_BIN="/opt/homebrew/bin/docker"
|
||||
GOOSED_COMPOSE="${GOOSED_DIR}/docker-compose.prod.yml"
|
||||
EDGE_MOBILE_BACKUP="${EDGE_MOBILE_CONFIG}.before-canary-${RELEASE_ID}"
|
||||
EDGE_WECHAT_BACKUP="${EDGE_WECHAT_CONFIG}.before-canary-${RELEASE_ID}"
|
||||
|
||||
say() {
|
||||
printf '\n[remote %s] %s\n' "$(date +%H:%M:%S)" "$*"
|
||||
@@ -239,36 +260,70 @@ candidate_healthy() {
|
||||
return 1
|
||||
}
|
||||
|
||||
write_tunnel_plist() {
|
||||
local local_port="$1"
|
||||
cat > "${TUNNEL_PLIST}" <<EOF
|
||||
edge_ssh() {
|
||||
# This remote release script itself arrives on stdin. Ordinary nested SSH
|
||||
# commands must not consume the unparsed remainder of that script.
|
||||
ssh -n -o BatchMode=yes -o ConnectTimeout=10 "${EDGE_HOST}" "$@"
|
||||
}
|
||||
|
||||
write_canary_tunnel_plist() {
|
||||
cat > "${CANARY_TUNNEL_PLIST}" <<EOF
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key><string>${PORTAL_TUNNEL_LABEL}</string>
|
||||
<key>Label</key><string>${CANARY_TUNNEL_LABEL}</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array><string>${STABLE_DIR}/scripts/memind-portal-tunnel.sh</string></array>
|
||||
<key>EnvironmentVariables</key>
|
||||
<dict>
|
||||
<key>MEMIND_PORTAL_TUNNEL_HOST</key><string>${MEMIND_PORTAL_TUNNEL_HOST}</string>
|
||||
<key>MEMIND_PORTAL_TUNNEL_LOCAL_PORT</key><string>${local_port}</string>
|
||||
<key>MEMIND_PORTAL_TUNNEL_REMOTE_PORT</key><string>${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}</string>
|
||||
<key>MEMIND_PORTAL_TUNNEL_HOST</key><string>${EDGE_HOST}</string>
|
||||
<key>MEMIND_PORTAL_TUNNEL_LOCAL_PORT</key><string>${CANARY_PROXY_PORT}</string>
|
||||
<key>MEMIND_PORTAL_TUNNEL_REMOTE_PORT</key><string>${CANARY_TUNNEL_REMOTE_PORT}</string>
|
||||
</dict>
|
||||
<key>RunAtLoad</key><true/>
|
||||
<key>KeepAlive</key><true/>
|
||||
<key>StandardOutPath</key><string>${HOME}/Library/Logs/memind-portal-tunnel.log</string>
|
||||
<key>StandardErrorPath</key><string>${HOME}/Library/Logs/memind-portal-tunnel.log</string>
|
||||
<key>StandardOutPath</key><string>${HOME}/Library/Logs/memind-canary-tunnel.log</string>
|
||||
<key>StandardErrorPath</key><string>${HOME}/Library/Logs/memind-canary-tunnel.log</string>
|
||||
</dict>
|
||||
</plist>
|
||||
EOF
|
||||
launchctl bootout "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null 2>&1 || true
|
||||
launchctl bootstrap "${LAUNCHD_GUI}" "${TUNNEL_PLIST}" >/dev/null
|
||||
launchctl enable "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null 2>&1 || true
|
||||
launchctl kickstart -k "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null
|
||||
launchctl bootout "${LAUNCHD_GUI}/${CANARY_TUNNEL_LABEL}" >/dev/null 2>&1 || true
|
||||
launchctl bootstrap "${LAUNCHD_GUI}" "${CANARY_TUNNEL_PLIST}" >/dev/null
|
||||
launchctl enable "${LAUNCHD_GUI}/${CANARY_TUNNEL_LABEL}" >/dev/null 2>&1 || true
|
||||
launchctl kickstart -k "${LAUNCHD_GUI}/${CANARY_TUNNEL_LABEL}" >/dev/null
|
||||
}
|
||||
|
||||
restore_edge_to_stable() {
|
||||
edge_ssh \
|
||||
"set -euo pipefail
|
||||
mobile_backup_exists=0
|
||||
wechat_backup_exists=0
|
||||
[[ -f '${EDGE_MOBILE_BACKUP}' ]] && mobile_backup_exists=1
|
||||
[[ -f '${EDGE_WECHAT_BACKUP}' ]] && wechat_backup_exists=1
|
||||
[[ \"\${mobile_backup_exists}\" == \"\${wechat_backup_exists}\" ]]
|
||||
if [[ \"\${mobile_backup_exists}\" -eq 1 ]]; then
|
||||
test -f '${EDGE_MOBILE_BACKUP}.sha256'
|
||||
sha256sum -c '${EDGE_MOBILE_BACKUP}.sha256' >/dev/null
|
||||
cp '${EDGE_MOBILE_BACKUP}' '${EDGE_MOBILE_CONFIG}'
|
||||
cp '${EDGE_WECHAT_BACKUP}' '${EDGE_WECHAT_CONFIG}'
|
||||
fi
|
||||
grep -q 'proxy_pass http://58.38.22.103:8081;' '${EDGE_MOBILE_CONFIG}'
|
||||
grep -q 'proxy_pass http://58.38.22.103:8081;' '${EDGE_WECHAT_CONFIG}'
|
||||
! grep -q 'proxy_pass http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT};' '${EDGE_MOBILE_CONFIG}'
|
||||
! grep -q 'proxy_pass http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT};' '${EDGE_WECHAT_CONFIG}'
|
||||
nginx -t >/dev/null
|
||||
systemctl reload nginx
|
||||
curl -kfsS --max-time 15 \
|
||||
--resolve m.tkmind.cn:443:127.0.0.1 \
|
||||
https://m.tkmind.cn/api/status >/dev/null
|
||||
curl -kfsS --max-time 15 \
|
||||
--resolve wechat.m.tkmind.cn:443:127.0.0.1 \
|
||||
https://wechat.m.tkmind.cn/api/status >/dev/null"
|
||||
}
|
||||
|
||||
stop_candidate_services() {
|
||||
launchctl bootout "${LAUNCHD_GUI}/${CANARY_TUNNEL_LABEL}" >/dev/null 2>&1 || true
|
||||
launchctl bootout "${LAUNCHD_GUI}/${CANARY_PROXY_LABEL}" >/dev/null 2>&1 || true
|
||||
launchctl bootout "${LAUNCHD_GUI}/${PORTAL_CANDIDATE_LABEL}" >/dev/null 2>&1 || true
|
||||
"${DOCKER_BIN}" rm -f goosed-prod-canary >/dev/null 2>&1 || true
|
||||
@@ -277,10 +332,14 @@ stop_candidate_services() {
|
||||
rollback() {
|
||||
local status=$?
|
||||
trap - ERR
|
||||
say "Canary rollout failed; restoring stable-only tunnel"
|
||||
say "Canary rollout failed; restoring the direct stable edge upstream"
|
||||
rm -f "${STABLE_DIR}/.release-drain"
|
||||
write_tunnel_plist 8081 >/dev/null 2>&1 || true
|
||||
stop_candidate_services
|
||||
if restore_edge_to_stable >/dev/null 2>&1; then
|
||||
stop_candidate_services
|
||||
else
|
||||
say "CRITICAL: edge restore could not be verified; keeping canary services running"
|
||||
say "Run scripts/rollback-portal-canary-prod.sh after checking the 105 nginx configs"
|
||||
fi
|
||||
exit "${status}"
|
||||
}
|
||||
trap rollback ERR
|
||||
@@ -323,9 +382,21 @@ shasum -a 256 -c "${PERSIST_BACKUP_TAR}.sha256"
|
||||
gzip -t "${PERSIST_BACKUP_TAR}"
|
||||
tar -tzf "${PERSIST_BACKUP_TAR}" | grep -E '^(\./)?\.env$' >/dev/null
|
||||
|
||||
if [[ -f "${TUNNEL_PLIST}" ]]; then
|
||||
cp "${TUNNEL_PLIST}" "${TUNNEL_PLIST_BACKUP}"
|
||||
fi
|
||||
say "Create and verify the active 105 nginx routing backup"
|
||||
edge_ssh \
|
||||
"set -euo pipefail
|
||||
test -f '${EDGE_MOBILE_CONFIG}'
|
||||
test -f '${EDGE_WECHAT_CONFIG}'
|
||||
grep -q 'proxy_pass http://58.38.22.103:8081;' '${EDGE_MOBILE_CONFIG}'
|
||||
grep -q 'proxy_pass http://58.38.22.103:8081;' '${EDGE_WECHAT_CONFIG}'
|
||||
! grep -q 'proxy_pass http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT};' '${EDGE_MOBILE_CONFIG}'
|
||||
! grep -q 'proxy_pass http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT};' '${EDGE_WECHAT_CONFIG}'
|
||||
cp '${EDGE_MOBILE_CONFIG}' '${EDGE_MOBILE_BACKUP}'
|
||||
cp '${EDGE_WECHAT_CONFIG}' '${EDGE_WECHAT_BACKUP}'
|
||||
sha256sum '${EDGE_MOBILE_BACKUP}' '${EDGE_WECHAT_BACKUP}' \
|
||||
> '${EDGE_MOBILE_BACKUP}.sha256'
|
||||
sha256sum -c '${EDGE_MOBILE_BACKUP}.sha256'
|
||||
nginx -t >/dev/null"
|
||||
|
||||
say "Extract the source-free candidate runtime"
|
||||
rm -rf "${CANDIDATE_DIR}"
|
||||
@@ -337,10 +408,7 @@ chmod 755 \
|
||||
"${CANDIDATE_DIR}/scripts/run-memind-canary-proxy-prod.sh"
|
||||
|
||||
say "Stop an older canary without touching stable Portal 8081"
|
||||
if curl -fsS http://127.0.0.1:18080/api/status >/dev/null 2>&1; then
|
||||
write_tunnel_plist 8081
|
||||
sleep 2
|
||||
fi
|
||||
restore_edge_to_stable
|
||||
stop_candidate_services
|
||||
|
||||
say "Start an isolated goosed candidate on 18015"
|
||||
@@ -416,7 +484,7 @@ chmod 600 "${SECRET_FILE}"
|
||||
/opt/homebrew/opt/node@24/bin/node "${CANDIDATE_DIR}/memind-canary-proxy.mjs" --check
|
||||
)
|
||||
|
||||
say "Start the fail-closed identity router on 18080"
|
||||
say "Start the fail-closed identity router on 18082"
|
||||
cat > "${PROXY_PLIST}" <<EOF
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
@@ -429,6 +497,7 @@ cat > "${PROXY_PLIST}" <<EOF
|
||||
<dict>
|
||||
<key>MEMIND_CANARY_STABLE_ROOT</key><string>${STABLE_DIR}</string>
|
||||
<key>MEMIND_CANARY_SECRET_FILE</key><string>${SECRET_FILE}</string>
|
||||
<key>MEMIND_CANARY_PROXY_PORT</key><string>${CANARY_PROXY_PORT}</string>
|
||||
<key>MEMIND_RELEASE_CANARY_USERNAMES</key><string>${CANARY_USERNAMES}</string>
|
||||
<key>MEMIND_RELEASE_CANARY_WECHAT_USER_IDS</key><string>${CANARY_WECHAT_USER_IDS}</string>
|
||||
</dict>
|
||||
@@ -447,7 +516,7 @@ diagnostic_secret="$(tr -d '\r\n' < "${SECRET_FILE}")"
|
||||
diagnostic="$(
|
||||
curl -fsS \
|
||||
-H "X-Memind-Canary-Secret: ${diagnostic_secret}" \
|
||||
http://127.0.0.1:18080/__memind_canary/health
|
||||
"http://127.0.0.1:${CANARY_PROXY_PORT}/__memind_canary/health"
|
||||
)"
|
||||
printf '%s' "${diagnostic}" | /opt/homebrew/opt/node@24/bin/node --input-type=module -e '
|
||||
let input = "";
|
||||
@@ -462,7 +531,7 @@ for probe_query in 'username=john' 'wechat_user_id=wx_ul610et8'; do
|
||||
route_probe="$(
|
||||
curl -fsS \
|
||||
-H "X-Memind-Canary-Secret: ${diagnostic_secret}" \
|
||||
"http://127.0.0.1:18080/__memind_canary/route-probe?${probe_query}"
|
||||
"http://127.0.0.1:${CANARY_PROXY_PORT}/__memind_canary/route-probe?${probe_query}"
|
||||
)"
|
||||
printf '%s' "${route_probe}" | /opt/homebrew/opt/node@24/bin/node --input-type=module -e '
|
||||
let input = "";
|
||||
@@ -475,7 +544,21 @@ for probe_query in 'username=john' 'wechat_user_id=wx_ul610et8'; do
|
||||
'
|
||||
done
|
||||
|
||||
say "Drain active Agent runs for the bounded tunnel switch"
|
||||
say "Start the isolated 105 reverse tunnel on 19082"
|
||||
write_canary_tunnel_plist
|
||||
for _ in $(seq 1 30); do
|
||||
edge_code="$(
|
||||
edge_ssh \
|
||||
"curl -sS -o /dev/null -w '%{http_code}' \
|
||||
--max-time 5 http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT}/api/status" \
|
||||
2>/dev/null || true
|
||||
)"
|
||||
[[ "${edge_code}" == "200" ]] && break
|
||||
sleep 2
|
||||
done
|
||||
[[ "${edge_code:-}" == "200" ]]
|
||||
|
||||
say "Drain active Agent runs for the bounded edge switch"
|
||||
touch "${STABLE_DIR}/.release-drain"
|
||||
deadline=$(( $(date +%s) + 120 ))
|
||||
while (( $(date +%s) < deadline )); do
|
||||
@@ -504,15 +587,33 @@ while (( $(date +%s) < deadline )); do
|
||||
done
|
||||
[[ "${counts:-unknown unknown}" == "0 0" ]]
|
||||
|
||||
say "Switch only the reverse tunnel from stable 8081 to identity router 18080"
|
||||
write_tunnel_plist 18080
|
||||
sleep 2
|
||||
remote_code="$(
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=10 "${MEMIND_PORTAL_TUNNEL_HOST}" \
|
||||
"curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}/api/status" \
|
||||
2>/dev/null || true
|
||||
)"
|
||||
[[ "${remote_code}" == "200" ]]
|
||||
say "Switch the committed 105 nginx upstreams to the isolated canary tunnel"
|
||||
# Unlike ordinary edge_ssh calls, this command intentionally receives only the
|
||||
# bounded EDGE_SWITCH heredoc on stdin.
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=10 "${EDGE_HOST}" /bin/bash <<EDGE_SWITCH
|
||||
set -euo pipefail
|
||||
for config in '${EDGE_MOBILE_CONFIG}' '${EDGE_WECHAT_CONFIG}'; do
|
||||
before_count="\$(grep -cF 'proxy_pass http://58.38.22.103:8081;' "\${config}")"
|
||||
[[ "\${before_count}" -gt 0 ]]
|
||||
temp_config="\$(mktemp "\${config}.canary.XXXXXX")"
|
||||
sed 's#proxy_pass http://58\.38\.22\.103:8081;#proxy_pass http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT};#g' \
|
||||
"\${config}" > "\${temp_config}"
|
||||
after_count="\$(grep -cF 'proxy_pass http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT};' "\${temp_config}")"
|
||||
[[ "\${after_count}" == "\${before_count}" ]]
|
||||
chown root:root "\${temp_config}"
|
||||
chmod 0644 "\${temp_config}"
|
||||
mv "\${temp_config}" "\${config}"
|
||||
done
|
||||
nginx -t
|
||||
systemctl reload nginx
|
||||
curl -fsS --max-time 10 http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT}/api/status >/dev/null
|
||||
curl -kfsS --max-time 15 \
|
||||
--resolve m.tkmind.cn:443:127.0.0.1 \
|
||||
https://m.tkmind.cn/api/status >/dev/null
|
||||
curl -kfsS --max-time 15 \
|
||||
--resolve wechat.m.tkmind.cn:443:127.0.0.1 \
|
||||
https://wechat.m.tkmind.cn/api/status >/dev/null
|
||||
EDGE_SWITCH
|
||||
rm -f "${STABLE_DIR}/.release-drain"
|
||||
|
||||
say "Verify candidate failure automatically falls back to stable"
|
||||
@@ -526,7 +627,7 @@ done
|
||||
fallback_probe="$(
|
||||
curl -fsS \
|
||||
-H "X-Memind-Canary-Secret: ${diagnostic_secret}" \
|
||||
'http://127.0.0.1:18080/__memind_canary/route-probe?username=john'
|
||||
"http://127.0.0.1:${CANARY_PROXY_PORT}/__memind_canary/route-probe?username=john"
|
||||
)"
|
||||
printf '%s' "${fallback_probe}" | /opt/homebrew/opt/node@24/bin/node --input-type=module -e '
|
||||
let input = "";
|
||||
@@ -537,6 +638,9 @@ printf '%s' "${fallback_probe}" | /opt/homebrew/opt/node@24/bin/node --input-typ
|
||||
if (status.candidate !== false || status.route !== "stable") process.exit(1);
|
||||
});
|
||||
'
|
||||
edge_ssh \
|
||||
"curl -fsS --max-time 10 \
|
||||
http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT}/api/status >/dev/null"
|
||||
launchctl bootstrap "${LAUNCHD_GUI}" "${CANDIDATE_PLIST}" >/dev/null
|
||||
launchctl kickstart -k "${LAUNCHD_GUI}/${PORTAL_CANDIDATE_LABEL}" >/dev/null
|
||||
for _ in $(seq 1 60); do
|
||||
@@ -547,7 +651,7 @@ candidate_healthy
|
||||
final_diagnostic="$(
|
||||
curl -fsS \
|
||||
-H "X-Memind-Canary-Secret: ${diagnostic_secret}" \
|
||||
http://127.0.0.1:18080/__memind_canary/health
|
||||
"http://127.0.0.1:${CANARY_PROXY_PORT}/__memind_canary/health"
|
||||
)"
|
||||
printf '%s' "${final_diagnostic}" | /opt/homebrew/opt/node@24/bin/node --input-type=module -e '
|
||||
let input = "";
|
||||
@@ -564,8 +668,11 @@ printf 'release_id=%s\n' "${RELEASE_ID}"
|
||||
printf 'candidate_dir=%s\n' "${CANDIDATE_DIR}"
|
||||
printf 'full_backup=%s\n' "${FULL_BACKUP_TAR}"
|
||||
printf 'persist_backup=%s\n' "${PERSIST_BACKUP_TAR}"
|
||||
printf 'edge_mobile_backup=%s:%s\n' "${EDGE_HOST}" "${EDGE_MOBILE_BACKUP}"
|
||||
printf 'edge_wechat_backup=%s:%s\n' "${EDGE_HOST}" "${EDGE_WECHAT_BACKUP}"
|
||||
printf 'stable_health=http://127.0.0.1:8081/api/status\n'
|
||||
printf 'proxy_health=http://127.0.0.1:18080/__memind_canary/health\n'
|
||||
printf 'proxy_health=http://127.0.0.1:%s/__memind_canary/health\n' "${CANARY_PROXY_PORT}"
|
||||
printf 'edge_tunnel=http://127.0.0.1:%s/api/status\n' "${CANARY_TUNNEL_REMOTE_PORT}"
|
||||
printf 'candidate_health=http://127.0.0.1:18081/api/status\n'
|
||||
printf 'candidate_goosed=https://127.0.0.1:18015/status\n'
|
||||
REMOTE_SCRIPT
|
||||
|
||||
@@ -5,9 +5,11 @@ HOST="${STUDIO_HOST:-58.38.22.103}"
|
||||
STABLE_DIR="${STUDIO_REMOTE_ROOT:-/Users/john/Project}/Memind"
|
||||
PORTAL_CANDIDATE_LABEL="cn.tkmind.memind-portal-candidate"
|
||||
CANARY_PROXY_LABEL="cn.tkmind.memind-canary-proxy"
|
||||
PORTAL_TUNNEL_LABEL="cn.tkmind.memind-portal-tunnel"
|
||||
MEMIND_PORTAL_TUNNEL_HOST="${MEMIND_PORTAL_TUNNEL_HOST:-ssh105-public}"
|
||||
MEMIND_PORTAL_TUNNEL_REMOTE_PORT="${MEMIND_PORTAL_TUNNEL_REMOTE_PORT:-19081}"
|
||||
CANARY_TUNNEL_LABEL="cn.tkmind.memind-canary-tunnel"
|
||||
EDGE_HOST="${MEMIND_CANARY_EDGE_HOST:-ssh105-public}"
|
||||
CANARY_TUNNEL_REMOTE_PORT=19082
|
||||
EDGE_MOBILE_CONFIG="/etc/nginx/conf.d/m.tkmind.cn.conf"
|
||||
EDGE_WECHAT_CONFIG="/etc/nginx/conf.d/wechat.m.tkmind.cn.conf"
|
||||
AUTO_YES=0
|
||||
|
||||
if [[ "${1:-}" == "--yes" || "${1:-}" == "-y" ]]; then
|
||||
@@ -18,7 +20,7 @@ elif [[ $# -gt 0 ]]; then
|
||||
fi
|
||||
|
||||
if [[ "${AUTO_YES}" -ne 1 ]]; then
|
||||
echo "This restores the 105 tunnel to stable Portal 8081 and stops only canary services."
|
||||
echo "This restores both 105 nginx upstreams to stable Portal 8081 and stops only canary services."
|
||||
read -r -p "Continue with 103 canary rollback? [y/N] " confirm </dev/tty
|
||||
[[ "${confirm}" =~ ^[Yy]$ ]] || exit 0
|
||||
fi
|
||||
@@ -27,55 +29,52 @@ ssh -o BatchMode=yes -o ConnectTimeout=15 "${HOST}" \
|
||||
"STABLE_DIR='${STABLE_DIR}' \
|
||||
PORTAL_CANDIDATE_LABEL='${PORTAL_CANDIDATE_LABEL}' \
|
||||
CANARY_PROXY_LABEL='${CANARY_PROXY_LABEL}' \
|
||||
PORTAL_TUNNEL_LABEL='${PORTAL_TUNNEL_LABEL}' \
|
||||
MEMIND_PORTAL_TUNNEL_HOST='${MEMIND_PORTAL_TUNNEL_HOST}' \
|
||||
MEMIND_PORTAL_TUNNEL_REMOTE_PORT='${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}' \
|
||||
CANARY_TUNNEL_LABEL='${CANARY_TUNNEL_LABEL}' \
|
||||
EDGE_HOST='${EDGE_HOST}' \
|
||||
CANARY_TUNNEL_REMOTE_PORT='${CANARY_TUNNEL_REMOTE_PORT}' \
|
||||
EDGE_MOBILE_CONFIG='${EDGE_MOBILE_CONFIG}' \
|
||||
EDGE_WECHAT_CONFIG='${EDGE_WECHAT_CONFIG}' \
|
||||
/bin/bash" <<'REMOTE'
|
||||
set -euo pipefail
|
||||
|
||||
LAUNCHD_GUI="gui/$(id -u)"
|
||||
TUNNEL_PLIST="${HOME}/Library/LaunchAgents/${PORTAL_TUNNEL_LABEL}.plist"
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=10 "${EDGE_HOST}" /bin/bash <<EDGE_ROLLBACK
|
||||
set -euo pipefail
|
||||
for config in '${EDGE_MOBILE_CONFIG}' '${EDGE_WECHAT_CONFIG}'; do
|
||||
canary_count="\$(grep -cF 'proxy_pass http://127.0.0.1:${CANARY_TUNNEL_REMOTE_PORT};' "\${config}" || true)"
|
||||
stable_count_before="\$(grep -cF 'proxy_pass http://58.38.22.103:8081;' "\${config}" || true)"
|
||||
if [[ "\${canary_count}" -eq 0 ]]; then
|
||||
[[ "\${stable_count_before}" -gt 0 ]]
|
||||
continue
|
||||
fi
|
||||
temp_config="\$(mktemp "\${config}.stable.XXXXXX")"
|
||||
sed 's#proxy_pass http://127\.0\.0\.1:${CANARY_TUNNEL_REMOTE_PORT};#proxy_pass http://58.38.22.103:8081;#g' \
|
||||
"\${config}" > "\${temp_config}"
|
||||
stable_count="\$(grep -cF 'proxy_pass http://58.38.22.103:8081;' "\${temp_config}")"
|
||||
[[ "\${stable_count}" -eq \$(( stable_count_before + canary_count )) ]]
|
||||
chown root:root "\${temp_config}"
|
||||
chmod 0644 "\${temp_config}"
|
||||
mv "\${temp_config}" "\${config}"
|
||||
done
|
||||
nginx -t
|
||||
systemctl reload nginx
|
||||
curl -kfsS --max-time 15 \
|
||||
--resolve m.tkmind.cn:443:127.0.0.1 \
|
||||
https://m.tkmind.cn/api/status >/dev/null
|
||||
curl -kfsS --max-time 15 \
|
||||
--resolve wechat.m.tkmind.cn:443:127.0.0.1 \
|
||||
https://wechat.m.tkmind.cn/api/status >/dev/null
|
||||
EDGE_ROLLBACK
|
||||
|
||||
cat > "${TUNNEL_PLIST}" <<EOF
|
||||
<?xml version="1.0" encoding="UTF-8"?>
|
||||
<!DOCTYPE plist PUBLIC "-//Apple//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">
|
||||
<plist version="1.0">
|
||||
<dict>
|
||||
<key>Label</key><string>${PORTAL_TUNNEL_LABEL}</string>
|
||||
<key>ProgramArguments</key>
|
||||
<array><string>${STABLE_DIR}/scripts/memind-portal-tunnel.sh</string></array>
|
||||
<key>EnvironmentVariables</key>
|
||||
<dict>
|
||||
<key>MEMIND_PORTAL_TUNNEL_HOST</key><string>${MEMIND_PORTAL_TUNNEL_HOST}</string>
|
||||
<key>MEMIND_PORTAL_TUNNEL_LOCAL_PORT</key><string>8081</string>
|
||||
<key>MEMIND_PORTAL_TUNNEL_REMOTE_PORT</key><string>${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}</string>
|
||||
</dict>
|
||||
<key>RunAtLoad</key><true/>
|
||||
<key>KeepAlive</key><true/>
|
||||
<key>StandardOutPath</key><string>${HOME}/Library/Logs/memind-portal-tunnel.log</string>
|
||||
<key>StandardErrorPath</key><string>${HOME}/Library/Logs/memind-portal-tunnel.log</string>
|
||||
</dict>
|
||||
</plist>
|
||||
EOF
|
||||
|
||||
launchctl bootout "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null 2>&1 || true
|
||||
launchctl bootstrap "${LAUNCHD_GUI}" "${TUNNEL_PLIST}" >/dev/null
|
||||
launchctl kickstart -k "${LAUNCHD_GUI}/${PORTAL_TUNNEL_LABEL}" >/dev/null
|
||||
launchctl bootout "${LAUNCHD_GUI}/${CANARY_TUNNEL_LABEL}" >/dev/null 2>&1 || true
|
||||
launchctl bootout "${LAUNCHD_GUI}/${CANARY_PROXY_LABEL}" >/dev/null 2>&1 || true
|
||||
launchctl bootout "${LAUNCHD_GUI}/${PORTAL_CANDIDATE_LABEL}" >/dev/null 2>&1 || true
|
||||
/opt/homebrew/bin/docker rm -f goosed-prod-canary >/dev/null 2>&1 || true
|
||||
rm -f "${STABLE_DIR}/.release-drain"
|
||||
|
||||
curl -fsS http://127.0.0.1:8081/api/status >/dev/null
|
||||
sleep 2
|
||||
remote_code="$(
|
||||
ssh -o BatchMode=yes -o ConnectTimeout=10 "${MEMIND_PORTAL_TUNNEL_HOST}" \
|
||||
"curl -s -o /dev/null -w '%{http_code}' http://127.0.0.1:${MEMIND_PORTAL_TUNNEL_REMOTE_PORT}/api/status" \
|
||||
2>/dev/null || true
|
||||
)"
|
||||
[[ "${remote_code}" == "200" ]]
|
||||
printf 'stable_port=8081\n'
|
||||
printf 'remote_tunnel_status=%s\n' "${remote_code}"
|
||||
printf 'edge_upstream=58.38.22.103:8081\n'
|
||||
REMOTE
|
||||
|
||||
echo "103 canary rollback completed; stable Portal remains active on 8081."
|
||||
|
||||
@@ -23,7 +23,7 @@ export NODE_ENV=production
|
||||
export MEMIND_PORTAL_H5_ROOT="${STABLE_ROOT}"
|
||||
export MEMIND_CANARY_DIAGNOSTIC_SECRET="$(tr -d '\r\n' < "${SECRET_FILE}")"
|
||||
export MEMIND_CANARY_PROXY_HOST=127.0.0.1
|
||||
export MEMIND_CANARY_PROXY_PORT="${MEMIND_CANARY_PROXY_PORT:-18080}"
|
||||
export MEMIND_CANARY_PROXY_PORT="${MEMIND_CANARY_PROXY_PORT:-18082}"
|
||||
export MEMIND_CANARY_STABLE_URL="${MEMIND_CANARY_STABLE_URL:-http://127.0.0.1:8081}"
|
||||
export MEMIND_CANARY_CANDIDATE_URL="${MEMIND_CANARY_CANDIDATE_URL:-http://127.0.0.1:18081}"
|
||||
|
||||
|
||||
Reference in New Issue
Block a user