feat: require validation receipts for h5 code runs
This commit is contained in:
Your Name
@@ -19,12 +19,12 @@
|
||||
- Stream Controller 已具备 SSE headers、abort propagation、backpressure pipeline。
|
||||
- Redis Router 已启用，承担 worker runtime state。
|
||||
- Goose Worker Pool 已从固定单点走向四 worker 可观测调度。
|
||||
- Aider/OpenHands 已从普通聊天默认能力中剥离，进入 code mode 和后端灰度门禁；P6.3 已把 code run 从 goosed session extension 外移到 `agent-run-v1` Tool Gateway 协议，P6.4 已完成 Aider 真实执行 canary，P6.5 已完成 OpenHands 真实执行 canary，P6.6 已完成 external worker 精确接管 code-run canary，P6.7 已加入 Tool Gateway 产物校验与输出审计，P6.8 已安装 external worker LaunchAgent，P6.9 已完成带 validation 的 external worker 灰度 canary，P6.10 已加入 external worker 只读观测脚本，P6.11 已加入放量策略门禁，P6.12 已开启全用户长期灰度并通过普通测试用户真实路径，P6.13 已安装自动暂停 guard。
|
||||
- PG 和 MindSpace 仍保持生产数据边界，SLO 报告只做统计读取；P6.3-P6.13 不新增 schema migration，不删除或修改既有用户数据。
|
||||
- Aider/OpenHands 已从普通聊天默认能力中剥离，进入 code mode 和后端灰度门禁；P6.3 已把 code run 从 goosed session extension 外移到 `agent-run-v1` Tool Gateway 协议，P6.4 已完成 Aider 真实执行 canary，P6.5 已完成 OpenHands 真实执行 canary，P6.6 已完成 external worker 精确接管 code-run canary，P6.7 已加入 Tool Gateway 产物校验与输出审计，P6.8 已安装 external worker LaunchAgent，P6.9 已完成带 validation 的 external worker 灰度 canary，P6.10 已加入 external worker 只读观测脚本，P6.11 已加入放量策略门禁，P6.12 已开启全用户长期灰度并通过普通测试用户真实路径，P6.13 已安装自动暂停 guard，P6.15 已让 H5 code-run 自动补 receipt validation 并恢复后端 required validation。
|
||||
- PG 和 MindSpace 仍保持生产数据边界，SLO 报告只做统计读取；P6.3-P6.15 不新增 schema migration，不删除或修改既有用户数据。
|
||||
|
||||
整体执行评分: 9.95 / 10。
|
||||
|
||||
可以支撑当前 H5 streaming 稳定性改造的基础目标。自动采样、worker sidecar heartbeat、SLO 只读快照、SLO 日报定时器、SLO 日报保留策略、first-token EWMA、first-token p50/p95 窗口趋势、Tool Gateway Queue v0、外部 worker 接管入口、真实 worker canary、后端 code-mode canary、code-run 用户级灰度 gate、H5 页面编辑 UI canary、P6.3 Tool Gateway 协议化、P6.4 Aider 真实 canary、P6.5 OpenHands 真实 canary、P6.6 external worker code-run canary、P6.7 Tool Gateway guardrails、P6.8 worker LaunchAgent、P6.9 validated external worker canary、P6.10 worker observability、P6.11 rollout policy gates、P6.12 all-user gray 和 P6.13 auto-pause guard 已经落地。主要剩余差距转为更细粒度的任务级产物规范、灰度长期运行观测，以及后续是否提高 worker 并发。
|
||||
可以支撑当前 H5 streaming 稳定性改造的基础目标。自动采样、worker sidecar heartbeat、SLO 只读快照、SLO 日报定时器、SLO 日报保留策略、first-token EWMA、first-token p50/p95 窗口趋势、Tool Gateway Queue v0、外部 worker 接管入口、真实 worker canary、后端 code-mode canary、code-run 用户级灰度 gate、H5 页面编辑 UI canary、P6.3 Tool Gateway 协议化、P6.4 Aider 真实 canary、P6.5 OpenHands 真实 canary、P6.6 external worker code-run canary、P6.7 Tool Gateway guardrails、P6.8 worker LaunchAgent、P6.9 validated external worker canary、P6.10 worker observability、P6.11 rollout policy gates、P6.12 all-user gray、P6.13 auto-pause guard 和 P6.15 H5 validation metadata 已经落地。主要剩余差距转为更细粒度的任务级产物规范、用户可见进度/失败说明，以及后续是否提高 worker 并发。
|
||||
|
||||
## 实测结果
|
||||
|
||||
@@ -852,22 +852,34 @@ Data boundary:
|
||||
- 影响: 已按用户要求作为历史内容问题忽略，不阻断 streaming runtime 发布。
|
||||
- 建议: 不在本轮修复；后续只有用户重新要求时再单独处理。
|
||||
|
||||
### P6.15 H5 Code-run Validation Metadata
|
||||
|
||||
结果: 通过，H5 默认 code-run 已自动补 receipt validation，生产后端 required validation 已恢复。
|
||||
|
||||
- 前端自动生成 `.memind/agent-runs/<requestId>.json` receipt validation。
|
||||
- `createAgentRun()` 会把 validation 写入 `metadata.memindRun.validation`，并给执行器追加 agent-visible receipt 指令。
|
||||
- 生产已恢复 `MEMIND_AGENT_CODE_RUNS_REQUIRE_VALIDATION=1`。
|
||||
- 普通测试用户真实 run 已成功:
|
||||
- run id `5eb28014-b376-4a0c-b42a-8dd38f781f4f`
|
||||
- request id `p615-validation-20260702034524`
|
||||
- attempts `1`
|
||||
- event chain 包含 `tool_gateway_validation`
|
||||
- SLO `ok=true`、`failures=[]`，guard `shouldPause=false`。
|
||||
|
||||
## 下一步执行建议
|
||||
|
||||
### P6.14 Gray Ops Observation
|
||||
### P6.16 Task Artifact UX and Failure Messages
|
||||
|
||||
- 观察 30-60 分钟:
|
||||
- `MEMIND_AGENT_RUN_WORKER_EXPECT_RUNNING=1 node scripts/check-agent-run-worker.mjs`
|
||||
- `node scripts/agent-run-guard.mjs --dry-run`
|
||||
- `node scripts/runtime-slo-report.mjs`
|
||||
- `tail -200 ~/Library/Logs/memind-agent-run-worker.log`
|
||||
- `tail -200 ~/Library/Logs/memind-agent-run-guard.log`
|
||||
- 重点判断:
|
||||
- queue 是否持续为空或快速归零。
|
||||
- 是否出现新的 failed run。
|
||||
- guard 是否误触发或正确保持 `shouldPause=false`。
|
||||
- SLO 是否仍 `failures=[]`。
|
||||
- 观察稳定后再评估:
|
||||
- 是否保持 worker 单并发。
|
||||
- 是否把 validation metadata 自动补入 H5 默认 code-run。
|
||||
- 是否增加任务级产物规范和用户可见进度/失败说明。
|
||||
- 为 H5 code-run 增加用户可见状态:
|
||||
- queued / running / validation / succeeded / failed
|
||||
- 显示“正在执行代码任务”而不是普通连接态。
|
||||
- 失败原因分层:
|
||||
- 执行器失败
|
||||
- 产物校验失败
|
||||
- 超时
|
||||
- 队列积压或 guard 暂停
|
||||
- 产物规范从 receipt 扩展到任务级:
|
||||
- 页面编辑任务声明目标 public HTML。
|
||||
- 仓库/文件任务声明目标相对路径。
|
||||
- 无明确目标的通用 code-run 继续使用 receipt 兜底。
|
||||
- P6.16 完成并稳定后，再评估 worker 并发是否从 1 提到 2。
|
||||
|
||||
@@ -23,6 +23,7 @@
|
||||
- P6.11 Single-user Rollout Policy: 已完成默认关闭的 code-run 放量策略门禁，支持 user allowlist、task type allowlist、required validation，并在 runtime/status 暴露当前策略。
|
||||
- P6.12 All-user Long-running Gray: 已完成，生产已默认对所有用户开启 H5 code-run 灰度，external worker running，普通测试用户真实 code-run 已成功生成 MindSpace 产物。
|
||||
- P6.13 Agent Run Auto-pause Guard: 已完成，新增 `agent-run-guard.mjs` 和 LaunchAgent，按失败数/队列积压/running 超时自动关闭 code-run gate 并停 worker。
|
||||
- P6.15 H5 Code-run Validation Metadata: 已完成，H5 code-run 自动声明 `.memind/agent-runs/<requestId>.json` receipt 校验，生产已恢复 `MEMIND_AGENT_CODE_RUNS_REQUIRE_VALIDATION=1` 并通过普通测试用户真实路径。
|
||||
- P5.15 Active Stream TTL Reconcile: 已按用户要求跳过，暂不做报表/定时 reconcile。
|
||||
- P5 Worker Pool 运维化: 已完成第一步，Redis Router 支持 worker drain。
|
||||
- P5.9 First-token Latency EWMA: 已完成，StreamController 会把首个 SSE chunk 延迟写入 Redis，SLO 报告已展示。
|
||||
@@ -2579,6 +2580,67 @@ runtime/status:
|
||||
|
||||
- `/Users/john/Project/memind_backups/20260702-113106-agent-run-guard-install`
|
||||
|
||||
### 2026-07-02 P6.15 H5 Code-run Validation Metadata
|
||||
|
||||
目标:
|
||||
|
||||
- 在 P6.14 灰度稳定后，把 H5 默认 code-run 从“允许无 validation”升级为“自动带 validation”。
|
||||
- 恢复后端 required validation gate，避免全用户灰度下出现无产物校验的 code-run。
|
||||
- 不猜测用户任务真正产物；统一要求每次 code-run 额外写入可验证 receipt。
|
||||
|
||||
新增行为:
|
||||
|
||||
- H5 code-run 创建时自动生成:
|
||||
- receipt path: `.memind/agent-runs/<requestId>.json`
|
||||
- validation: `metadata.memindRun.validation.expectedFile`
|
||||
- contains: 当前 `requestId`
|
||||
- H5 会把简短校验指令追加到 agent-visible message:
|
||||
- 要求执行器创建/更新 receipt JSON。
|
||||
- receipt 必须包含当前 requestId 值。
|
||||
- `createAgentRun()` 在提交 `/api/agent/runs` 前合并 validation metadata。
|
||||
- 普通 chat mode 不附加 receipt validation。
|
||||
|
||||
生产配置:
|
||||
|
||||
- 已恢复:
|
||||
- `MEMIND_AGENT_CODE_RUNS_REQUIRE_VALIDATION=1`
|
||||
- 保持:
|
||||
- `MEMIND_AGENT_CODE_RUNS_ENABLED=1`
|
||||
- `MEMIND_AGENT_CODE_RUNS_USER_IDS=`
|
||||
- `MEMIND_AGENT_CODE_RUN_TASK_TYPES=`
|
||||
- `MEMIND_AGENT_RUN_AUTODISPATCH=0`
|
||||
- external worker running
|
||||
- auto-pause guard enabled
|
||||
|
||||
验证:
|
||||
|
||||
- `npm run build` 通过。
|
||||
- `node --test agent-run-routes.test.mjs agent-run-gateway.test.mjs` 通过，27 tests pass。
|
||||
- 生产 `/api/runtime/status`:
|
||||
- `codeRunPolicy.enabled=true`
|
||||
- `codeRunPolicy.userAllowlist=[]`
|
||||
- `codeRunPolicy.taskTypeAllowlist=[]`
|
||||
- `codeRunPolicy.requireValidation=true`
|
||||
- 普通测试用户真实路径通过:
|
||||
- user id `52ff79cf-21b2-4185-94d6-f24555126ceb`
|
||||
- run id `5eb28014-b376-4a0c-b42a-8dd38f781f4f`
|
||||
- request id `p615-validation-20260702034524`
|
||||
- status `succeeded`
|
||||
- attempts `1`
|
||||
- 产物验证:
|
||||
- receipt: `/Users/john/Project/Memind/MindSpace/52ff79cf-21b2-4185-94d6-f24555126ceb/.memind/agent-runs/p615-validation-20260702034524.json`
|
||||
- product: `/Users/john/Project/Memind/MindSpace/52ff79cf-21b2-4185-94d6-f24555126ceb/public/p615-validation-20260702034524.html`
|
||||
- event chain 包含 `tool_gateway_validation`。
|
||||
- SLO:
|
||||
- `ok=true`
|
||||
- `failures=[]`
|
||||
- queue empty
|
||||
- guard `shouldPause=false`
|
||||
|
||||
备份:
|
||||
|
||||
- `/Users/john/Project/memind_backups/20260702-114335-p615-validation-metadata`
|
||||
|
||||
## 回滚策略
|
||||
|
||||
- P0: 修改前保留 `server.mjs` 备份；如启动失败，恢复备份并 `launchctl kickstart` Portal。
|
||||
|
||||
+43
-2
@@ -58,7 +58,7 @@ import type {
|
||||
} from '../types';
|
||||
import { CHAT_IMAGE_UPLOAD_MAX_INPUT_BYTES } from '../utils/imageUpload';
|
||||
import { normalizeConversationMessages, normalizeUserMessageForApi } from '../utils/message';
|
||||
import type { AgentRunCreateOptions } from '../utils/agentRunMode';
|
||||
import type { AgentRunCreateOptions, AgentRunValidation } from '../utils/agentRunMode';
|
||||
|
||||
const API = '/api';
|
||||
const DEFAULT_API_TIMEOUT_MS = 20_000;
|
||||
@@ -79,6 +79,46 @@ export type AgentRun = {
|
||||
completedAt: number | null;
|
||||
};
|
||||
|
||||
function appendAgentRunValidationInstruction(message: Message, instruction?: string | null): Message {
|
||||
const normalizedInstruction = String(instruction ?? '').trim();
|
||||
if (!normalizedInstruction) return message;
|
||||
return {
|
||||
...message,
|
||||
content: [
|
||||
...message.content,
|
||||
{ type: 'text', text: normalizedInstruction },
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
function withAgentRunValidationMetadata(
|
||||
message: Message,
|
||||
validation?: AgentRunValidation | null,
|
||||
): Message {
|
||||
if (!validation) return message;
|
||||
const metadata = message.metadata as Message['metadata'] & {
|
||||
memindRun?: Record<string, unknown>;
|
||||
};
|
||||
return {
|
||||
...message,
|
||||
metadata: {
|
||||
...message.metadata,
|
||||
memindRun: {
|
||||
...(metadata.memindRun && typeof metadata.memindRun === 'object' ? metadata.memindRun : {}),
|
||||
validation,
|
||||
},
|
||||
} as Message['metadata'],
|
||||
};
|
||||
}
|
||||
|
||||
function prepareAgentRunUserMessage(message: Message, options: AgentRunCreateOptions): Message {
|
||||
const normalized = normalizeUserMessageForApi(message);
|
||||
return appendAgentRunValidationInstruction(
|
||||
withAgentRunValidationMetadata(normalized, options.validation),
|
||||
options.toolMode === 'code' ? options.validationInstruction : null,
|
||||
);
|
||||
}
|
||||
|
||||
export class ApiError extends Error {
|
||||
readonly status: number;
|
||||
readonly code?: string;
|
||||
@@ -2250,6 +2290,7 @@ export async function createAgentRun(
|
||||
userMessage: Message,
|
||||
options: AgentRunCreateOptions = {},
|
||||
): Promise<AgentRun> {
|
||||
const userMessagePayload = prepareAgentRunUserMessage(userMessage, options);
|
||||
const result = await apiFetch<{ run: AgentRun }>(
|
||||
AGENT_RUNS_PATH,
|
||||
{
|
||||
@@ -2257,7 +2298,7 @@ export async function createAgentRun(
|
||||
body: JSON.stringify({
|
||||
session_id: sessionId,
|
||||
request_id: requestId,
|
||||
user_message: normalizeUserMessageForApi(userMessage),
|
||||
user_message: userMessagePayload,
|
||||
...(options.toolMode ? { tool_mode: options.toolMode } : {}),
|
||||
...(options.taskType ? { task_type: options.taskType } : {}),
|
||||
}),
|
||||
|
||||
@@ -348,6 +348,7 @@ export function usePageEditSubChat({
|
||||
taskType: 'page_edit_code_task',
|
||||
forceCode: true,
|
||||
userId: user?.id ?? null,
|
||||
requestId,
|
||||
}),
|
||||
);
|
||||
const finishedRun =
|
||||
|
||||
@@ -1176,6 +1176,7 @@ export function useTKMindChat(
|
||||
resolveAgentRunOptions(trimmed, {
|
||||
taskType: 'h5_chat_code_task',
|
||||
userId: userRef.current?.id ?? null,
|
||||
requestId,
|
||||
}),
|
||||
);
|
||||
const finishedRun =
|
||||
|
||||
@@ -1,6 +1,18 @@
|
||||
export type AgentRunCreateOptions = {
|
||||
toolMode?: 'chat' | 'code';
|
||||
taskType?: string | null;
|
||||
validation?: AgentRunValidation | null;
|
||||
validationInstruction?: string | null;
|
||||
};
|
||||
|
||||
export type AgentRunValidationFile = {
|
||||
path: string;
|
||||
contains?: string;
|
||||
};
|
||||
|
||||
export type AgentRunValidation = {
|
||||
expectedFile?: AgentRunValidationFile | string;
|
||||
expectedFiles?: Array<AgentRunValidationFile | string>;
|
||||
};
|
||||
|
||||
function envFlag(value: unknown): boolean {
|
||||
@@ -38,6 +50,36 @@ const CODE_TASK_PATTERNS = [
|
||||
/(代码|仓库|项目|文件|组件|接口).{0,12}(修改|修复|重构|调试|实现|新增|编写|更新)/,
|
||||
];
|
||||
|
||||
function sanitizeRequestIdForPath(requestId: string): string {
|
||||
const normalized = String(requestId ?? '').trim().replace(/[^a-zA-Z0-9._-]/g, '-');
|
||||
return normalized || 'unknown-request';
|
||||
}
|
||||
|
||||
export function buildAgentRunValidationReceipt(requestId: string): {
|
||||
validation: AgentRunValidation;
|
||||
instruction: string;
|
||||
} {
|
||||
const normalizedRequestId = String(requestId ?? '').trim();
|
||||
const safeRequestId = sanitizeRequestIdForPath(normalizedRequestId);
|
||||
const receiptPath = `.memind/agent-runs/${safeRequestId}.json`;
|
||||
const marker = normalizedRequestId || safeRequestId;
|
||||
return {
|
||||
validation: {
|
||||
expectedFile: {
|
||||
path: receiptPath,
|
||||
contains: marker,
|
||||
},
|
||||
},
|
||||
instruction: [
|
||||
'',
|
||||
'[Memind code-run validation]',
|
||||
`Before finishing, create or update ${receiptPath}.`,
|
||||
`The file must be valid JSON and include this requestId value: ${marker}.`,
|
||||
'Do not skip this receipt even if the user task itself is complete.',
|
||||
].join('\n'),
|
||||
};
|
||||
}
|
||||
|
||||
export function resolveAgentRunOptions(
|
||||
text: string,
|
||||
{
|
||||
@@ -45,19 +87,24 @@ export function resolveAgentRunOptions(
|
||||
forceCode = false,
|
||||
allowAutodetect = agentCodeRunsAutodetectEnabled,
|
||||
userId = null,
|
||||
requestId = null,
|
||||
}: {
|
||||
taskType?: string;
|
||||
forceCode?: boolean;
|
||||
allowAutodetect?: boolean;
|
||||
userId?: string | null;
|
||||
requestId?: string | null;
|
||||
} = {},
|
||||
): AgentRunCreateOptions {
|
||||
if (!agentCodeRunsEnabledForUser(userId)) return {};
|
||||
const normalizedText = String(text ?? '').trim();
|
||||
const shouldUseCode = forceCode || (allowAutodetect && CODE_TASK_PATTERNS.some((pattern) => pattern.test(normalizedText)));
|
||||
if (!shouldUseCode) return {};
|
||||
const receipt = buildAgentRunValidationReceipt(requestId ?? crypto.randomUUID());
|
||||
return {
|
||||
toolMode: 'code',
|
||||
taskType,
|
||||
validation: receipt.validation,
|
||||
validationInstruction: receipt.instruction,
|
||||
};
|
||||
}
|
||||
|
||||
Reference in New Issue
Block a user