Compare commits

..

15 Commits

Author SHA1 Message Date
john 2f4366e882 merge: enable WeChat media for all users
Memind CI / Test, build, and release guards (push) Successful in 2m18s
2026-07-18 21:21:13 +08:00
john 2ce1527edd feat: enable WeChat media for all users 2026-07-18 21:20:56 +08:00
john 1161292482 merge: fix WeChat multi-image gray flow
Memind CI / Test, build, and release guards (push) Successful in 2m28s
2026-07-18 19:53:21 +08:00
john 711a7d2061 fix: aggregate WeChat multi-image batches 2026-07-18 19:53:09 +08:00
john 341cf5ae89 merge: fix WeChat media gray pipeline
Memind CI / Test, build, and release guards (push) Successful in 1m55s
2026-07-18 19:10:13 +08:00
john ae3e1ba3fa fix: recover stale WeChat active requests 2026-07-18 19:05:33 +08:00
john 8eaf4d23f3 fix: route WeChat media through shared vision pipeline 2026-07-18 18:15:22 +08:00
john 8ccaf3b39d merge: add WeChat media analysis gray rollout
Memind CI / Test, build, and release guards (push) Successful in 2m35s
2026-07-18 17:50:06 +08:00
john 94f347398d feat: gate WeChat media analysis by user 2026-07-18 17:46:20 +08:00
john fd3904fdee feat: add WeChat media analysis adapters 2026-07-18 17:41:32 +08:00
tkmind 055d53c58b Merge pull request 'fix(chat): translate session replay cursors' (#14)
Memind CI / Test, build, and release guards (push) Successful in 4m33s
Preserve Portal/Goose SSE cursor boundaries and recover Finish after reconnect.
2026-07-17 08:55:36 +00:00
john cce56a4c6a fix(chat): translate session replay cursors
Memind CI / Test, build, and release guards (pull_request) Successful in 2m55s
2026-07-17 16:49:31 +08:00
tkmind c9591bc4cf Merge pull request 'fix(analytics): identify Memind users before pageviews' (#13) from codex/fix-umami-users-identify into main
Memind CI / Test, build, and release guards (push) Successful in 4m25s
2026-07-17 06:17:00 +00:00
john ed9e23fc07 fix(analytics): identify Memind users before pageviews
Memind CI / Test, build, and release guards (pull_request) Successful in 3m11s
2026-07-17 14:10:36 +08:00
tkmind 25620be0b1 Merge pull request 'fix: sync npm lockfile for Excel runtime' (#12) from codex/fix-excel-package-lock into main
Memind CI / Test, build, and release guards (push) Successful in 3m39s
Keep package-lock.json aligned with the Excel runtime dependency so npm ci and release CI remain reproducible.
2026-07-17 04:16:20 +00:00
23 changed files with 1453 additions and 60 deletions
+3
View File
@@ -29,6 +29,7 @@ bash scripts/check-release-ready.sh
| MindSpace 公开页 `edit_file` 落盘 + 聊天 Finish 不丢消息 | [docs/regression-guards/mindspace-publish-and-chat-finish.md](docs/regression-guards/mindspace-publish-and-chat-finish.md) |
| MindSpace remote 页面 sync + storage 缺失缩略图 fallback | [docs/regression-guards/mindspace-remote-page-sync-and-thumbnail.md](docs/regression-guards/mindspace-remote-page-sync-and-thumbnail.md) |
| Page Data 数据集注册、绑定与交付验收 | [docs/regression-guards/page-data-delivery-contract.md](docs/regression-guards/page-data-delivery-contract.md) |
| H5 SSE 断线续播、Portal/Goose 游标映射与 Finish 终态恢复 | [docs/regression-guards/h5-session-stream-replay.md](docs/regression-guards/h5-session-stream-replay.md) |
索引:[docs/regression-guards/README.md](docs/regression-guards/README.md)
@@ -38,6 +39,7 @@ bash scripts/check-release-ready.sh
npm run verify:mindspace-publish-guards
npm run verify:mindspace-publish-guards:full
npm run verify:mindspace-page-sync-guards
npm run verify:h5-session-patches
```
发版脚本(`scripts/release-portal-runtime-prod.sh`)在未 `--skip-tests` 时也会执行相关 verify。
@@ -51,6 +53,7 @@ npm run verify:mindspace-page-sync-guards
- `server.mjs` - session snapshot 需 `hint_mc``hint_ua` 才走缓存
- `mindspace-pages.mjs` - storage 缺失时 HTML 页回退读 workspace `relative_path`
- `mindspace-page-sync-service.mjs` + `server.mjs` - remote 模式也必须 sync public HTML
- `session-stream.mjs` + `session-stream-store.mjs` + `tkmind-proxy.mjs` - Portal replay ID 不得直接作为 Goose `Last-Event-ID`
代码内搜索 `REGRESSION GUARD` 可定位所有内联说明。
+5 -1
View File
@@ -24,7 +24,11 @@ local Umami service at `http://127.0.0.1:3100`; it does not contact 103/105.
```
4. Restart the local Memind server. Full generated HTML pages will receive a
same-origin `/analytics/script.js` tracker and a `page_view` event with
same-origin `/analytics/script.js` tracker. The tracker identifies the
visitor with a stable pseudonymous owner ID before sending a standard Umami
page view, so Users and Pageviews are populated. The Identify properties
include the readable Memind username and current public page URL for
operational analytics. Click, form, scroll, and engagement events retain the
pseudonymous `owner_id`, `page_id`, and `channel` dimensions.
The integration is fail-open: missing configuration, disabled analytics, or a
+2
View File
@@ -9,12 +9,14 @@
| [mindspace-publish-and-chat-finish.md](./mindspace-publish-and-chat-finish.md) | ① `edit_file` 覆盖 `public/*.html` ② Finish 后聊天不清空、不暴露 agent 内部前缀 |
| [mindspace-remote-page-sync-and-thumbnail.md](./mindspace-remote-page-sync-and-thumbnail.md) | ① remote 模式 public HTML 入库 sync ② storage 缺失时缩略图/读页回退 workspace HTML |
| [page-data-delivery-contract.md](./page-data-delivery-contract.md) | 数据集注册、绑定、真实 page UUID 与交付验收 |
| [h5-session-stream-replay.md](./h5-session-stream-replay.md) | Portal/Goose SSE 游标映射、断线续播与 Finish 终态恢复 |
## 自动化
```bash
npm run verify:mindspace-publish-guards # 推荐:改相关代码后
npm run verify:mindspace-publish-guards:full # 发 Portal runtime 后
npm run verify:h5-session-patches # H5 会话、Agent Run 与 SSE 续播
```
## 新增守卫时
@@ -0,0 +1,40 @@
# H5 会话 SSE 断线续播守卫
## 症状
后台 `h5_agent_runs` 已经是 `succeeded`、Goose 也已发送 `Finish`,但 H5 输入区仍显示“正在执行任务…”。常见触发条件是手机切后台、锁屏或网络切换发生在 `ActiveRequests``Finish` 之间。
## 根因
Portal 对 SSE 帧使用本地持久化游标。该游标可能是 Portal 生成的 UUID,并不一定是 Goose 能识别的上游事件 ID。若重连时把 Portal UUID 原样作为 `Last-Event-ID` 发送给 Goose,Goose 无法续播,浏览器就收不到遗漏的 `Finish`
## 必须保留的行为
1. Portal 游标只用于查询 `h5_session_stream_events`,不得原样传给 Goose。
2. 连接 Goose 时,只能使用已持久化的 `upstream_event_id`
3. 当前游标或重放尾部没有可用 `upstream_event_id` 时,必须省略 `Last-Event-ID`,让 Goose 从权威事件流重新回放。
4. 本地重放最后一条已经是 `Finish``Error` 时,不再连接 Goose。
5. 重放不得破坏现有消息 ID 合并、Finish 幂等计费和 MindSpace Finish 同步守卫。
## 关键路径
- `session-stream.mjs`
- `session-stream-store.mjs`
- `tkmind-proxy.mjs`
- `src/hooks/useTKMindChat.ts`
## 回归测试
```bash
node --test session-stream.test.mjs session-stream-store.test.mjs chat-agent-run-gate.test.mjs
node --test --test-name-pattern='proxySessionEvents' tkmind-proxy.test.mjs
npm run verify:h5-session-patches
node --test billing-session-concurrency.test.mjs
```
必须覆盖:
- Portal 游标存在但 `upstream_event_id` 为空时,不向 Goose 发送错误游标,并能收到 `Finish`
- 有映射时,Portal 游标正确转换成 Goose 游标。
- 多轮会话中选择最新可映射的上游游标,不被上一轮 Finish 截断。
- Finish 重放不重复扣费,消息合并与页面同步守卫继续通过。
+5 -5
View File
@@ -95,10 +95,10 @@ export function injectMindSpaceAnalytics(html, {
if (source.includes(ANALYTICS_MARKER)) return source;
const owner = pseudonymizeAnalyticsId(ownerId, config.idSecret);
if (!owner) return source;
// Public page source must not contain a readable account name. The stable
// pseudonym and coarse plan segment are sufficient for page analytics;
// readable labels are reserved for server-originated events only.
const metadata = { owner_id: owner, owner_segment: String(ownerSegment || 'unknown'), page_id: String(pageId || ''), publication_id: String(publicationId || ''), channel };
// The stable pseudonym remains the Umami identity key. The readable username
// is an explicitly enabled analytics property so operators can recognize the
// Memind user, while the current public page URL is resolved in the browser.
const metadata = { owner_id: owner, username: resolveAnalyticsOwnerLabel({ displayName: ownerLabel }), owner_segment: String(ownerSegment || 'unknown'), page_id: String(pageId || ''), publication_id: String(publicationId || ''), channel };
const attrs = [
ANALYTICS_MARKER,
`data-website-id="${config.websiteId.replaceAll('"', '"')}"`,
@@ -106,7 +106,7 @@ export function injectMindSpaceAnalytics(html, {
`data-host-url="${config.hostPath}"`,
];
if (config.domains) attrs.push(`data-domains="${config.domains.replaceAll('"', '"')}"`);
const block = `<script ${attrs.join(' ')} defer src="${config.scriptPath}"></script><script ${ANALYTICS_MARKER}>(function(){var d=${jsonForInlineScript(metadata)},seen={};function t(n,x){if(!window.umami||typeof window.umami.track!=='function')return;var p=Object.assign({},d,{page_url:location.href,page_title:document.title},x||{});window.umami.track(n,p);}function once(n,x){if(seen[n])return;seen[n]=1;t(n,x);}function ready(){t('page_view');document.addEventListener('click',function(e){var el=e.target&&e.target.closest?e.target.closest('a,button,[role="button"],[data-umami-event]'):null;if(!el)return;var custom=el.getAttribute('data-umami-event');var href=el.tagName==='A'?el.getAttribute('href')||'':'';t(custom||'page_click',{element:el.tagName.toLowerCase(),element_id:el.id||'',event_label:(custom||el.getAttribute('aria-label')||'').slice(0,100),target_url:href.slice(0,500)});},{passive:true});document.addEventListener('submit',function(e){var form=e.target;t('page_form_submit',{form_id:form&&form.id||'',form_action:form&&form.getAttribute('action')||''});},{passive:true});var marks=[25,50,75,90];function scroll(){var h=document.documentElement.scrollHeight-window.innerHeight;if(h<=0){once('page_scroll_100');return;}var pct=Math.round(window.scrollY/h*100);marks.forEach(function(m){if(pct>=m)once('page_scroll_'+m);});}window.addEventListener('scroll',scroll,{passive:true});setTimeout(function(){once('page_engaged_10s');},10000);setTimeout(function(){once('page_engaged_30s');},30000);scroll();}if(document.readyState==='loading'){document.addEventListener('DOMContentLoaded',ready,{once:true});}else{ready();}})();</script>`;
const block = `<script ${attrs.join(' ')} defer src="${config.scriptPath}"></script><script ${ANALYTICS_MARKER}>(function(){var d=${jsonForInlineScript(metadata)},seen={};function t(n,x){if(!window.umami||typeof window.umami.track!=='function')return;var p=Object.assign({},d,{page_url:location.href,page_title:document.title},x||{});window.umami.track(n,p);}function identify(){if(!window.umami||typeof window.umami.identify!=='function')return;window.umami.identify(d.owner_id,{username:d.username,memind_page_url:location.href,owner_segment:d.owner_segment,channel:d.channel});}function pageview(){if(!window.umami||typeof window.umami.track!=='function')return;window.umami.track();}function once(n,x){if(seen[n])return;seen[n]=1;t(n,x);}function ready(){identify();pageview();document.addEventListener('click',function(e){var el=e.target&&e.target.closest?e.target.closest('a,button,[role="button"],[data-umami-event]'):null;if(!el)return;var custom=el.getAttribute('data-umami-event');var href=el.tagName==='A'?el.getAttribute('href')||'':'';t(custom||'page_click',{element:el.tagName.toLowerCase(),element_id:el.id||'',event_label:(custom||el.getAttribute('aria-label')||'').slice(0,100),target_url:href.slice(0,500)});},{passive:true});document.addEventListener('submit',function(e){var form=e.target;t('page_form_submit',{form_id:form&&form.id||'',form_action:form&&form.getAttribute('action')||''});},{passive:true});var marks=[25,50,75,90];function scroll(){var h=document.documentElement.scrollHeight-window.innerHeight;if(h<=0){once('page_scroll_100');return;}var pct=Math.round(window.scrollY/h*100);marks.forEach(function(m){if(pct>=m)once('page_scroll_'+m);});}window.addEventListener('scroll',scroll,{passive:true});setTimeout(function(){once('page_engaged_10s');},10000);setTimeout(function(){once('page_engaged_30s');},30000);scroll();}if(document.readyState==='loading'){document.addEventListener('DOMContentLoaded',ready,{once:true});}else{ready();}})();</script>`;
if (/<\/head>/i.test(source)) return source.replace(/<\/head>/i, `${block}</head>`);
return source.replace(/<body\b/i, `${block}<body`);
}
+55 -1
View File
@@ -1,5 +1,6 @@
import test from 'node:test';
import assert from 'node:assert/strict';
import vm from 'node:vm';
import {
injectMindSpaceAnalytics,
@@ -64,10 +65,14 @@ test('injects one local same-origin tracker with page dimensions', () => {
assert.match(out, /src="\/analytics\/script\.js"/);
assert.match(out, /data-host-url="\/analytics"/);
assert.match(out, /data-auto-track="false"/);
assert.match(out, /window\.umami\.identify\(d\.owner_id,\{username:d\.username,memind_page_url:location\.href,owner_segment:d\.owner_segment,channel:d\.channel\}\)/);
assert.match(out, /function pageview\(\).*window\.umami\.track\(\)/);
assert.ok(out.indexOf('identify();pageview();') > 0);
assert.doesNotMatch(out, /t\('page_view'\)/);
assert.match(out, /page_id/);
assert.match(out, /owner_segment/);
assert.doesNotMatch(out, /owner_label/);
assert.doesNotMatch(out, /张三/);
assert.match(out, /"username":"张三"/);
assert.match(out, /page_click/);
assert.match(out, /page_form_submit/);
assert.match(out, /page_scroll_/);
@@ -76,6 +81,55 @@ test('injects one local same-origin tracker with page dimensions', () => {
assert.equal(injectMindSpaceAnalytics(out, { ownerId: 'user-123', config: { enabled: true, websiteId: 'local-website', idSecret: 'secret' } }), out);
});
test('identifies the pseudonymous owner before sending a standard page view', () => {
const out = injectMindSpaceAnalytics('<!doctype html><html><head></head><body></body></html>', {
ownerId: 'user-123',
ownerSegment: 'plan:pro',
ownerLabel: '张三',
channel: 'h5',
config: {
enabled: true,
websiteId: 'local-website',
idSecret: 'secret',
scriptPath: '/analytics/script.js',
hostPath: '/analytics',
},
});
const inlineScript = out.match(/<script data-memind-analytics="1">([\s\S]*?)<\/script>/)?.[1];
assert.ok(inlineScript);
const calls = [];
vm.runInNewContext(inlineScript, {
window: {
umami: {
identify: (...args) => calls.push(['identify', ...args]),
track: (...args) => calls.push(['track', ...args]),
},
innerHeight: 800,
scrollY: 0,
addEventListener: () => {},
},
document: {
readyState: 'complete',
title: 'Demo',
documentElement: { scrollHeight: 1600 },
addEventListener: () => {},
},
location: { href: 'https://m.tkmind.cn/MindSpace/demo/public/page.html' },
setTimeout: () => {},
});
assert.deepEqual(JSON.parse(JSON.stringify(calls)), [
['identify', pseudonymizeAnalyticsId('user-123', 'secret'), {
username: '张三',
memind_page_url: 'https://m.tkmind.cn/MindSpace/demo/public/page.html',
owner_segment: 'plan:pro',
channel: 'h5',
}],
['track'],
]);
});
test('does not alter non-full-html or disabled pages', () => {
const fragment = '<div>hello</div>';
assert.equal(injectMindSpaceAnalytics(fragment, { ownerId: 'u', config: { enabled: true, websiteId: 'w', idSecret: 's' } }), fragment);
+2
View File
@@ -820,6 +820,8 @@ async function bootstrapUserAuth() {
const target = await tkmindProxy.resolveTarget(sessionId);
return tkmindProxy.apiFetchTo(target, pathname, init);
},
submitSessionReply: ({ userId, sessionId, requestId, userMessage }) =>
tkmindProxy.submitSessionReplyForUser(userId, sessionId, requestId, userMessage),
scheduleService: process.env.H5_SCHEDULE_ENABLED === '1' ? scheduleService : null,
wechatScheduleLlmConfigService,
llmProviderService,
+18 -16
View File
@@ -8,6 +8,17 @@ function safeJsonParse(value, fallback = null) {
}
}
function mapEventRow(row) {
if (!row) return null;
return {
id: row.id,
eventType: row.event_type,
payload: safeJsonParse(row.payload_json, null),
upstreamEventId: row.upstream_event_id ?? null,
createdAt: Number(row.created_at),
};
}
export function createSessionStreamStore({ pool }) {
if (!pool) {
throw new Error('createSessionStreamStore requires pool');
@@ -52,9 +63,10 @@ export function createSessionStreamStore({ pool }) {
let afterCreatedAt = null;
let cursorMiss = false;
let cursorEvent = null;
if (afterEventId) {
const [cursorRows] = await pool.query(
`SELECT created_at
`SELECT id, event_type, payload_json, upstream_event_id, created_at
FROM h5_session_stream_events
WHERE id = ? AND agent_session_id = ? AND user_id = ?
LIMIT 1`,
@@ -63,7 +75,8 @@ export function createSessionStreamStore({ pool }) {
if (!cursorRows[0]) {
cursorMiss = true;
} else {
afterCreatedAt = Number(cursorRows[0].created_at);
cursorEvent = mapEventRow(cursorRows[0]);
afterCreatedAt = cursorEvent.createdAt;
}
}
@@ -85,14 +98,9 @@ export function createSessionStreamStore({ pool }) {
);
return {
events: rows.map((row) => ({
id: row.id,
eventType: row.event_type,
payload: safeJsonParse(row.payload_json, null),
upstreamEventId: row.upstream_event_id ?? null,
createdAt: Number(row.created_at),
})),
events: rows.map(mapEventRow),
cursorMiss,
cursorEvent,
};
}
@@ -107,13 +115,7 @@ export function createSessionStreamStore({ pool }) {
);
const row = rows[0];
if (!row) return null;
return {
id: row.id,
eventType: row.event_type,
payload: safeJsonParse(row.payload_json, null),
upstreamEventId: row.upstream_event_id ?? null,
createdAt: Number(row.created_at),
};
return mapEventRow(row);
}
return {
+6 -1
View File
@@ -23,7 +23,7 @@ function createMockPool() {
}
if (sql.includes('WHERE id = ? AND agent_session_id = ?')) {
const row = events.find((item) => item.id === params[0]);
return [[row ? { created_at: row.created_at } : undefined].filter(Boolean)];
return [[row].filter(Boolean)];
}
if (sql.includes('created_at > ?')) {
const after = Number(params[2]);
@@ -67,12 +67,14 @@ test('session stream store append and replay with cursor', async () => {
sessionId: 'sess-1',
payload: { type: 'Message', message: { role: 'assistant', content: [] } },
id: 'evt-1',
upstreamEventId: 'upstream-1',
});
await store.appendEvent({
userId: 'user-1',
sessionId: 'sess-1',
payload: { type: 'Finish' },
id: 'evt-2',
upstreamEventId: 'upstream-2',
});
const head = await store.listEventsForUser('user-1', 'sess-1');
@@ -82,7 +84,10 @@ test('session stream store append and replay with cursor', async () => {
assert.equal(tail.events.length, 1);
assert.equal(tail.events[0].payload.type, 'Finish');
assert.equal(tail.cursorMiss, false);
assert.equal(tail.cursorEvent.id, first.id);
assert.equal(tail.cursorEvent.upstreamEventId, 'upstream-1');
const miss = await store.listEventsForUser('user-1', 'sess-1', { afterEventId: 'missing' });
assert.equal(miss.cursorMiss, true);
assert.equal(miss.cursorEvent, null);
});
+20
View File
@@ -15,6 +15,26 @@ export function parseSessionStreamLastEventId(value) {
return id || null;
}
/**
* Portal replay ids are local database cursors and are not necessarily valid
* Goose SSE ids. Resume Goose from the newest persisted upstream cursor we
* actually know about. If none exists, return null so the caller omits
* Last-Event-ID and lets Goose replay the authoritative stream from the start.
*
* @param {Array<{ upstreamEventId?: string | null }>} events
* @param {{ upstreamEventId?: string | null } | null} cursorEvent
* @returns {string | null}
*/
export function resolveUpstreamResumeEventId(events, cursorEvent = null) {
const candidates = Array.isArray(events) ? [...events].reverse() : [];
if (cursorEvent) candidates.push(cursorEvent);
for (const event of candidates) {
const id = String(event?.upstreamEventId ?? '').trim();
if (id) return id;
}
return null;
}
export function isTerminalSessionEvent(event) {
const type = String(event?.type ?? '').trim();
return SESSION_STREAM_TERMINAL_TYPES.has(type);
+32
View File
@@ -6,6 +6,7 @@ import {
isTerminalSessionEvent,
parseSessionSseBlock,
parseSessionStreamLastEventId,
resolveUpstreamResumeEventId,
shouldPersistSessionStreamEvent,
shouldSkipUpstreamAfterSessionReplay,
} from './session-stream.mjs';
@@ -44,3 +45,34 @@ test('terminal detection and upstream skip policy', () => {
assert.equal(isSessionStreamReplayEnabled({ MEMIND_SESSION_STREAM_REPLAY: '1' }), true);
assert.equal(parseSessionStreamLastEventId(' abc '), 'abc');
});
test('upstream resume cursor never falls back to a Portal replay id', () => {
assert.equal(
resolveUpstreamResumeEventId([], {
id: 'portal-uuid',
upstreamEventId: null,
}),
null,
);
assert.equal(
resolveUpstreamResumeEventId([], {
id: 'portal-uuid',
upstreamEventId: 'upstream-7',
}),
'upstream-7',
);
});
test('upstream resume cursor uses the newest mappable replay event across turns', () => {
assert.equal(
resolveUpstreamResumeEventId(
[
{ upstreamEventId: 'upstream-old-finish' },
{ upstreamEventId: 'upstream-new-message' },
{ upstreamEventId: null },
],
{ upstreamEventId: 'upstream-cursor' },
),
'upstream-new-message',
);
});
+36
View File
@@ -83,6 +83,42 @@ test('buildVisionPayload marks one billable image analysis when vision succeeds'
);
});
test('buildVisionPayload sends all current-turn images to Qwen in order', async () => {
const imageUrls = [
'https://m.tkmind.cn/MindSpace/user-1/public/wechat-mp/first.jpg',
'https://m.tkmind.cn/MindSpace/user-1/public/wechat-mp/second.jpg',
];
let analyzedImages = [];
const result = await buildVisionPayload({
userId: 'user-1',
publishLayout: { publicUrl: 'https://m.tkmind.cn/MindSpace/user-1' },
userMessage: {
content: [{ type: 'text', text: '请结合两张图片生成页面' }],
metadata: { imageUrls },
},
fetchImpl: async () =>
new Response(Buffer.from('fake-image'), {
status: 200,
headers: { 'Content-Type': 'image/jpeg' },
}),
llmProviderService: {
analyzeImagesWithVision: async (images) => {
analyzedImages = images;
return '第一张是宝塔,第二张是晚霞。';
},
},
});
assert.equal(analyzedImages.length, 2);
assert.deepEqual(analyzedImages.map((item) => item.rawUrl), imageUrls);
assert.deepEqual(result?.userMessage?.metadata?.imageUrls, [
'/MindSpace/user-1/public/wechat-mp/first.jpg',
'/MindSpace/user-1/public/wechat-mp/second.jpg',
]);
assert.match(result?.userMessage?.content?.[0]?.text ?? '', /本轮用户仅上传 2 张图片/);
assert.match(result?.userMessage?.content?.[0]?.text ?? '', /图片2/);
});
test('buildVisionPayload does not mark billable usage when vision analysis fails', async () => {
const result = await buildVisionPayload({
userId: 'user-1',
+11 -4
View File
@@ -44,6 +44,7 @@ import {
isSessionStreamReplayEnabled,
parseSessionSseBlock,
parseSessionStreamLastEventId,
resolveUpstreamResumeEventId,
shouldPersistSessionStreamEvent,
shouldSkipUpstreamAfterSessionReplay,
} from './session-stream.mjs';
@@ -2154,6 +2155,7 @@ export function createTkmindProxy({
let clientClosed = false;
const replayEnabled = isSessionStreamReplayEnabled() && sessionStreamStore;
const initialLastEventId = parseSessionStreamLastEventId(req.get('last-event-id'));
let upstreamLastEventId = replayEnabled ? null : initialLastEventId;
const abortUpstream = () => {
clientClosed = true;
upstreamAbort.abort();
@@ -2210,17 +2212,22 @@ export function createTkmindProxy({
if (!res.writableEnded) res.end();
return;
}
upstreamLastEventId = resolveUpstreamResumeEventId(
batch?.events ?? [],
batch?.cursorEvent ?? null,
);
}
const pathname = `/sessions/${sessionId}/events`;
const sessionTarget = await resolveTarget(sessionId);
const upstreamHeaders = { Accept: 'text/event-stream' };
if (upstreamLastEventId) {
upstreamHeaders['Last-Event-ID'] = upstreamLastEventId;
}
const upstream = await apiFetch(sessionTarget, apiSecret, pathname, {
method: 'GET',
signal: upstreamAbort.signal,
headers: {
Accept: 'text/event-stream',
'Last-Event-ID': req.get('last-event-id') ?? '',
},
headers: upstreamHeaders,
});
if (clientClosed) return;
+233
View File
@@ -533,6 +533,178 @@ test('proxySessionEvents attaches session taxonomy when flag enabled', async ()
}
});
test('proxySessionEvents omits an unmapped Portal cursor so Goose can replay Finish', async () => {
const previousReplay = process.env.MEMIND_SESSION_STREAM_REPLAY;
process.env.MEMIND_SESSION_STREAM_REPLAY = '1';
let upstream;
let receivedLastEventId = 'not-requested';
try {
upstream = createServer((req, res) => {
if (req.method === 'GET' && req.url === '/sessions/session-1/events') {
receivedLastEventId = req.headers['last-event-id'] ?? null;
res.writeHead(200, { 'Content-Type': 'text/event-stream' });
res.end(
'id: upstream-finish\n' +
'data: {"type":"Finish","reason":"stop","request_id":"req-1"}\n\n',
);
return;
}
res.writeHead(404, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ message: `unexpected ${req.method} ${req.url}` }));
});
const upstreamPort = await listen(upstream);
const persisted = [];
const proxy = createTkmindProxy({
apiTarget: `http://127.0.0.1:${upstreamPort}`,
apiSecret: 'test-secret',
userAuth: createMemoryTestUserAuth(process.cwd()),
sessionStreamStore: {
async listEventsForUser(_userId, _sessionId, { afterEventId } = {}) {
assert.equal(afterEventId, 'portal-active-request-uuid');
return {
cursorMiss: false,
cursorEvent: {
id: 'portal-active-request-uuid',
payload: { type: 'ActiveRequests', request_ids: ['req-1'] },
upstreamEventId: null,
createdAt: 1,
},
events: [],
};
},
async appendEvent(event) {
persisted.push(event);
},
},
});
const req = new EventEmitter();
req.currentUser = { id: 'user-1', username: 'john' };
req.get = (name) =>
name.toLowerCase() === 'last-event-id' ? 'portal-active-request-uuid' : '';
req.once = req.once.bind(req);
req.off = req.off.bind(req);
const chunks = [];
const res = new EventEmitter();
res.headersSent = false;
res.writableEnded = false;
res.statusCode = 200;
res.status = (code) => {
res.statusCode = code;
return res;
};
res.setHeader = () => {};
res.flushHeaders = () => {
res.headersSent = true;
};
res.write = (chunk) => {
res.headersSent = true;
chunks.push(String(chunk));
return true;
};
res.end = () => {
res.writableEnded = true;
};
res.json = () => {
throw new Error('json must not be called after SSE started');
};
await proxy.proxySessionEvents(req, res, 'session-1');
assert.equal(receivedLastEventId, null);
assert.match(chunks.join(''), /"type":"Finish"/);
await new Promise((resolve) => setImmediate(resolve));
assert.equal(persisted.at(-1)?.upstreamEventId, 'upstream-finish');
} finally {
if (previousReplay == null) delete process.env.MEMIND_SESSION_STREAM_REPLAY;
else process.env.MEMIND_SESSION_STREAM_REPLAY = previousReplay;
await closeServer(upstream);
}
});
test('proxySessionEvents translates a Portal replay cursor to its Goose cursor', async () => {
const previousReplay = process.env.MEMIND_SESSION_STREAM_REPLAY;
process.env.MEMIND_SESSION_STREAM_REPLAY = '1';
let upstream;
let receivedLastEventId = null;
try {
upstream = createServer((req, res) => {
if (req.method === 'GET' && req.url === '/sessions/session-1/events') {
receivedLastEventId = req.headers['last-event-id'] ?? null;
res.writeHead(200, { 'Content-Type': 'text/event-stream' });
res.end('id: upstream-finish\ndata: {"type":"Finish","reason":"stop"}\n\n');
return;
}
res.writeHead(404, { 'Content-Type': 'application/json' });
res.end(JSON.stringify({ message: `unexpected ${req.method} ${req.url}` }));
});
const upstreamPort = await listen(upstream);
const proxy = createTkmindProxy({
apiTarget: `http://127.0.0.1:${upstreamPort}`,
apiSecret: 'test-secret',
userAuth: createMemoryTestUserAuth(process.cwd()),
sessionStreamStore: {
async listEventsForUser() {
return {
cursorMiss: false,
cursorEvent: {
id: 'portal-message-uuid',
payload: { type: 'Message' },
upstreamEventId: 'upstream-message-17',
createdAt: 1,
},
events: [],
};
},
async appendEvent() {},
},
});
const req = new EventEmitter();
req.currentUser = { id: 'user-1', username: 'john' };
req.get = (name) => name.toLowerCase() === 'last-event-id' ? 'portal-message-uuid' : '';
req.once = req.once.bind(req);
req.off = req.off.bind(req);
const chunks = [];
const res = new EventEmitter();
res.headersSent = false;
res.writableEnded = false;
res.statusCode = 200;
res.status = (code) => {
res.statusCode = code;
return res;
};
res.setHeader = () => {};
res.flushHeaders = () => {
res.headersSent = true;
};
res.write = (chunk) => {
res.headersSent = true;
chunks.push(String(chunk));
return true;
};
res.end = () => {
res.writableEnded = true;
};
res.json = () => {
throw new Error('json must not be called after SSE started');
};
await proxy.proxySessionEvents(req, res, 'session-1');
assert.equal(receivedLastEventId, 'upstream-message-17');
assert.match(chunks.join(''), /"type":"Finish"/);
} finally {
if (previousReplay == null) delete process.env.MEMIND_SESSION_STREAM_REPLAY;
else process.env.MEMIND_SESSION_STREAM_REPLAY = previousReplay;
await closeServer(upstream);
}
});
test('startSessionForUser resolves memories through Memory V2 facade', async () => {
let resolveInput = null;
await withFakeGoosedSession(async ({ apiTarget, workingDir, harnessEntries }) => {
@@ -726,6 +898,67 @@ test('submitSessionReplyForUser adds goose metadata visibility flags before repl
});
});
test('submitSessionReplyForUser applies the shared Qwen vision preprocessing path', async () => {
await withFakeGoosedSession(async ({ apiTarget, workingDir, replyBodies }) => {
const proxy = createTkmindProxy({
apiTarget,
apiSecret: 'test-secret',
userAuth: {
...createMemoryTestUserAuth(workingDir),
async ownsSession() {
return true;
},
async canUseChat() {
return { ok: true };
},
async getUserById() {
return { id: 'user-1' };
},
async getUserPublishLayout() {
return { publicUrl: 'https://example.com/MindSpace/user-1' };
},
async resolveUserPolicies() {
return { unrestricted: true, policies: {} };
},
},
localFetchAsset: async () => ({
buffer: Buffer.from('fake-image'),
mimeType: 'image/jpeg',
}),
llmProviderService: {
async applyBestProviderForSession() {
return { ok: true };
},
async hasVisionKey() {
return true;
},
async analyzeImagesWithVision() {
return '一件蓝色产品,白色背景,竖版构图。';
},
},
});
await proxy.submitSessionReplyForUser(
'user-1',
'session-1',
'request-qwen-vision',
{
id: 'message-qwen-vision',
role: 'user',
content: [{ type: 'text', text: '请分析这张图片' }],
metadata: {
imageUrls: ['/api/mindspace/v1/assets/asset-1/download?inline=1'],
displayText: '请分析这张图片',
},
},
);
const forwardedText = replyBodies[0]?.user_message?.content?.[0]?.text ?? '';
assert.match(forwardedText, /Qwen VL 图片描述/);
assert.match(forwardedText, /一件蓝色产品/);
});
});
test('submitSessionReplyForUser passes current prompt to Memory V2 resolve before existing reply path', async () => {
let resolveInput = null;
await withFakeGoosedSession(async ({ apiTarget, workingDir }) => {
+88
View File
@@ -9,12 +9,19 @@ const __dirname = path.dirname(fileURLToPath(import.meta.url));
const DEFAULT_WECHAT_MEDIA_URL = 'https://api.weixin.qq.com/cgi-bin/media/get';
const DEFAULT_MAX_IMAGE_BYTES = 10 * 1024 * 1024;
const DEFAULT_MAX_ATTACHMENT_BYTES = 30 * 1024 * 1024;
const ALLOWED_IMAGE_MIME_TYPES = new Map([
['image/jpeg', 'jpg'],
['image/png', 'png'],
['image/webp', 'webp'],
['image/gif', 'gif'],
]);
const ALLOWED_ATTACHMENT_EXTENSIONS = new Map([
['.doc', 'application/msword'],
['.docx', 'application/vnd.openxmlformats-officedocument.wordprocessingml.document'],
['.xls', 'application/vnd.ms-excel'],
['.xlsx', 'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet'],
]);
function resolveImageExtension(contentType = '', fallbackUrl = '') {
const normalized = String(contentType ?? '')
@@ -44,6 +51,32 @@ function ensureImageWithinLimit(buffer, maxBytes) {
}
}
function sanitizeAttachmentFilename(filename = '') {
const basename = path.basename(String(filename ?? '').trim()).replace(/[\u0000-\u001f\u007f]/g, '');
if (!basename || basename === '.' || basename === '..') {
throw new Error('微信文件缺少有效文件名');
}
const extension = path.extname(basename).toLowerCase();
const mimeType = ALLOWED_ATTACHMENT_EXTENSIONS.get(extension);
if (!mimeType) {
throw new Error('当前服务号文件仅支持 Worddoc/docx)和 Excelxls/xlsx');
}
return {
filename: basename.slice(0, 160),
extension,
mimeType,
};
}
function ensureAttachmentWithinLimit(buffer, maxBytes) {
if (!Buffer.isBuffer(buffer) || buffer.length === 0) {
throw new Error('微信文件内容为空');
}
if (buffer.length > maxBytes) {
throw new Error(`文件超过大小限制(${maxBytes} bytes`);
}
}
export async function downloadTemporaryMedia(accessToken, mediaId, { wechatFetch = undiciFetch } = {}) {
if (!accessToken) throw new Error('缺少微信 access_token');
if (!mediaId) throw new Error('缺少微信 mediaId');
@@ -154,3 +187,58 @@ export async function persistWechatImage(
source,
};
}
export async function persistWechatAttachment(
{
userId,
appId,
openid,
msgId,
mediaId,
filename,
publicBaseUrl,
maxFileBytes = DEFAULT_MAX_ATTACHMENT_BYTES,
},
{
wechatFetch = undiciFetch,
accessToken,
h5Root = __dirname,
} = {},
) {
if (!userId) throw new Error('缺少 userId');
const resolved = sanitizeAttachmentFilename(filename);
const downloaded = await downloadTemporaryMedia(accessToken, mediaId, { wechatFetch });
ensureAttachmentWithinLimit(downloaded.buffer, maxFileBytes);
const publishDir = path.join(h5Root, PUBLISH_ROOT_DIR, String(userId), PUBLIC_ZONE_DIR, 'wechat-mp');
fs.mkdirSync(publishDir, { recursive: true });
const timestamp = Date.now();
const hash = crypto.createHash('sha1').update(downloaded.buffer).digest('hex').slice(0, 12);
const originalStem = path.basename(resolved.filename, resolved.extension)
.replace(/[^\p{L}\p{N}._-]+/gu, '_')
.replace(/^_+|_+$/g, '')
.slice(0, 80) || 'attachment';
const identity = [appId || 'wx', openid || 'openid', msgId || timestamp, mediaId || hash]
.filter(Boolean)
.join('-')
.replace(/[^a-zA-Z0-9._-]+/g, '_')
.slice(0, 100);
const publicFilename = `${originalStem}-${identity}-${hash}${resolved.extension}`;
const absolutePath = path.join(publishDir, publicFilename);
fs.writeFileSync(absolutePath, downloaded.buffer);
return {
absolutePath,
bytes: downloaded.buffer.length,
contentType: resolved.mimeType,
filename: resolved.filename,
publicFilename,
publicUrl: buildWechatImagePublicUrl({
publicBaseUrl,
publishKey: String(userId),
filename: publicFilename,
}),
source: 'wechat_media',
};
}
+10
View File
@@ -23,6 +23,13 @@ function deriveWechatEndpointFromUrl(baseUrl, suffix) {
}
}
function parseCsvList(value = '') {
return String(value ?? '')
.split(',')
.map((item) => item.trim())
.filter(Boolean);
}
export function loadWechatMpConfig(env = process.env) {
const appId = env.H5_WECHAT_MP_APP_ID?.trim() ?? env.H5_WECHAT_APP_ID?.trim() ?? '';
const appSecret =
@@ -67,10 +74,13 @@ export function loadWechatMpConfig(env = process.env) {
mediaPublicBaseUrl:
env.H5_WECHAT_MP_MEDIA_PUBLIC_BASE_URL?.trim()?.replace(/\/$/, '') || publicBaseUrl,
maxImageBytes: Math.max(1, Number(env.H5_WECHAT_MP_MAX_IMAGE_BYTES ?? 10 * 1024 * 1024)),
maxFileBytes: Math.max(1, Number(env.H5_WECHAT_MP_MAX_FILE_BYTES ?? 30 * 1024 * 1024)),
acceptVoice: env.H5_WECHAT_MP_ACCEPT_VOICE !== '0',
acceptImage: env.H5_WECHAT_MP_ACCEPT_IMAGE !== '0',
acceptFile: env.H5_WECHAT_MP_ACCEPT_FILE !== '0',
acceptLocation: env.H5_WECHAT_MP_ACCEPT_LOCATION !== '0',
acceptLink: env.H5_WECHAT_MP_ACCEPT_LINK !== '0',
mediaAnalysisGrayUsers: parseCsvList(env.H5_WECHAT_MP_MEDIA_GRAY_USERS),
encodingAesKey: env.H5_WECHAT_MP_ENCODING_AES_KEY?.trim() ?? '',
};
}
+305 -30
View File
@@ -9,7 +9,11 @@ import { resolveSessionAccess } from './session-broker.mjs';
import { isStubPublicHtmlContent, materializeMissingPublicHtmlWrites } from './mindspace-public-finish-sync.mjs';
import { loadWechatMpConfig } from './wechat-mp-config.mjs';
import { buildPublicUrl, PUBLISH_ROOT_DIR } from './user-publish.mjs';
import { downloadTemporaryMedia, persistWechatImage } from './wechat-media.mjs';
import {
downloadTemporaryMedia,
persistWechatAttachment,
persistWechatImage,
} from './wechat-media.mjs';
import { normalizeWechatName, resolveWechatAddressName } from './wechat/user/display-name.mjs';
import { buildAckText } from './wechat/ack/ack-provider.mjs';
import { guardScheduleConfirmationReply } from './wechat/handlers/schedule-guard.mjs';
@@ -46,6 +50,8 @@ const DEFAULT_WECHAT_CUSTOMER_SERVICE_URL =
'https://api.weixin.qq.com/cgi-bin/message/custom/send';
const DEFAULT_WECHAT_JSAPI_TICKET_URL = 'https://api.weixin.qq.com/cgi-bin/ticket/getticket';
const DEFAULT_ASR_TARGET = process.env.H5_ASR_TARGET ?? 'https://asr.tkmind.cn';
const WECHAT_RECENT_MEDIA_TTL_MS = 15 * 60 * 1000;
const WECHAT_RECENT_IMAGE_MAX_COUNT = 10;
export { loadWechatMpConfig };
const PUBLIC_HTML_LINK_PATTERN =
/https?:\/\/[^\s<>"')\]]+\/MindSpace\/([0-9a-f-]{36}|[a-z0-9._-]+)\/public\/([^\s<>"')\]]+\.html)/gi;
@@ -100,6 +106,8 @@ function parseWechatMessage(xml) {
description: parseXmlField(xml, 'Description'),
url: parseXmlField(xml, 'Url'),
thumbMediaId: parseXmlField(xml, 'ThumbMediaId'),
fileName: parseXmlField(xml, 'FileName') || parseXmlField(xml, 'Filename'),
fileSize: parseXmlField(xml, 'FileSize'),
event: parseXmlField(xml, 'Event').toLowerCase(),
eventKey: parseXmlField(xml, 'EventKey'),
latitude: parseXmlField(xml, 'Latitude'),
@@ -204,7 +212,14 @@ function pushMessage(messages, incoming) {
return [...messages, incoming];
}
async function executeSessionReply(apiFetch, sessionId, requestId, prompt, metadata = {}) {
async function executeSessionReply(
apiFetch,
sessionId,
requestId,
prompt,
metadata = {},
{ submitReply = null } = {},
) {
const eventsResponse = await apiFetch(`/sessions/${sessionId}/events`, {
method: 'GET',
headers: { Accept: 'text/event-stream' },
@@ -214,18 +229,23 @@ async function executeSessionReply(apiFetch, sessionId, requestId, prompt, metad
throw new Error(text || '无法建立公众号消息事件流');
}
const replyResponse = await apiFetch(`/sessions/${sessionId}/reply`, {
method: 'POST',
body: JSON.stringify({
request_id: requestId,
user_message: createUserMessage(prompt, metadata),
}),
});
if (!replyResponse.ok) {
const text = await replyResponse.text().catch(() => '');
throw new Error(text || 'Agent reply 请求失败');
const userMessage = createUserMessage(prompt, metadata);
if (submitReply) {
await submitReply({ sessionId, requestId, userMessage });
} else {
const replyResponse = await apiFetch(`/sessions/${sessionId}/reply`, {
method: 'POST',
body: JSON.stringify({
request_id: requestId,
user_message: userMessage,
}),
});
if (!replyResponse.ok) {
const text = await replyResponse.text().catch(() => '');
throw new Error(text || 'Agent reply 请求失败');
}
replyResponse.body?.cancel().catch?.(() => {});
}
replyResponse.body?.cancel().catch?.(() => {});
const reader = eventsResponse.body.getReader();
const decoder = new TextDecoder();
@@ -994,6 +1014,8 @@ export function isRecoverableWechatAgentSessionError(message) {
if (/stale_session_poisoned_completion/i.test(normalized)) return true;
if (/403|404|not found|无权访问/i.test(normalized)) return true;
if (/tool_calls|tool_call_id|insufficient tool messages/i.test(normalized)) return true;
if (/session already has an active request|active request.*cancel/i.test(normalized)) return true;
if (/wechat_agent_incomplete_reply/i.test(normalized)) return true;
return false;
}
@@ -1024,6 +1046,13 @@ export function findRecoverableWechatAgentErrorInReply(reply) {
export function assertWechatAgentReplyIsSendable(reply) {
const recoverable = findRecoverableWechatAgentErrorInReply(reply);
if (recoverable) throw new Error(recoverable);
const text = String(reply?.text ?? '').trim();
if (
/^(?:let me|i(?:'ll| will))\s+(?:first\s+)?(?:look|check|inspect|analy[sz]e)(?:\s+at)?\s+(?:the\s+)?image(?:\s+first)?[.!]?$/iu.test(text) ||
/^(?:让我|我先)(?:先)?(?:看|查看|检查|分析)(?:一下)?(?:这张|该张|这个)?图片[。!!]?$/u.test(text)
) {
throw new Error('wechat_agent_incomplete_reply');
}
}
export function isWechatAgentApiErrorText(message) {
@@ -1164,6 +1193,24 @@ function normalizeNumber(value) {
return Number.isFinite(num) ? num : null;
}
function isWechatMediaGrayUser(user, configuredUsers = []) {
const allowlist = Array.isArray(configuredUsers)
? configuredUsers.map((value) => String(value ?? '').trim().toLowerCase()).filter(Boolean)
: [];
if (allowlist.length === 0) return false;
if (allowlist.includes('*')) return true;
const identities = [
user?.userId,
user?.username,
user?.slug,
user?.displayName,
user?.nickname,
]
.map((value) => String(value ?? '').trim().toLowerCase())
.filter(Boolean);
return identities.some((identity) => allowlist.includes(identity));
}
function normalizeWechatInboundIntent(inbound) {
const msgType = String(inbound?.msgType ?? '').toLowerCase();
const base = {
@@ -1210,6 +1257,22 @@ function normalizeWechatInboundIntent(inbound) {
};
}
if (msgType === 'file') {
const filename = String(inbound?.fileName ?? '').trim();
return {
...base,
displayText: filename ? `收到文件:${filename}` : '收到文件',
agentText: '',
media: {
mediaId: inbound?.mediaId || '',
},
attachment: {
filename,
sizeBytes: normalizeNumber(inbound?.fileSize),
},
};
}
if (msgType === 'location') {
const latitude = normalizeNumber(inbound?.locationX);
const longitude = normalizeNumber(inbound?.locationY);
@@ -1409,6 +1472,7 @@ export function createWechatMpService({
apiFetch,
startAgentSession = null,
sessionApiFetch = null,
submitSessionReply = null,
scheduleService = null,
wechatScheduleLlmConfigService = null,
llmProviderService = null,
@@ -1434,10 +1498,15 @@ export function createWechatMpService({
jsapiTicketUrl: config.jsapiTicketUrl || DEFAULT_WECHAT_JSAPI_TICKET_URL,
mediaPublicBaseUrl: config.mediaPublicBaseUrl || config.publicBaseUrl,
maxImageBytes: Math.max(1, Number(config.maxImageBytes ?? 10 * 1024 * 1024)),
maxFileBytes: Math.max(1, Number(config.maxFileBytes ?? 30 * 1024 * 1024)),
acceptVoice: config.acceptVoice !== false,
acceptImage: config.acceptImage !== false,
acceptFile: config.acceptFile !== false,
acceptLocation: config.acceptLocation !== false,
acceptLink: config.acceptLink !== false,
mediaAnalysisGrayUsers: Array.isArray(config.mediaAnalysisGrayUsers)
? config.mediaAnalysisGrayUsers
: [],
asrTarget: config.asrTarget || DEFAULT_ASR_TARGET,
};
@@ -1446,6 +1515,8 @@ export function createWechatMpService({
expiresAt: 0,
};
const rememberedWechatContexts = new Map();
const messageTasksByOpenid = new Map();
const recentMediaByOpenid = new Map();
let jsapiTicketCache = {
ticket: null,
expiresAt: 0,
@@ -1454,6 +1525,86 @@ export function createWechatMpService({
const fetchForSession = (sessionId, pathname, init) =>
sessionApiFetch ? sessionApiFetch(sessionId, pathname, init) : apiFetch(pathname, init);
const enqueueMessageTask = (openid, taskFactory) => {
const key = String(openid ?? '').trim();
const previous = messageTasksByOpenid.get(key) ?? Promise.resolve();
const task = previous.catch(() => undefined).then(taskFactory);
messageTasksByOpenid.set(key, task);
void task
.finally(() => {
if (messageTasksByOpenid.get(key) === task) messageTasksByOpenid.delete(key);
})
.catch(() => {});
return task;
};
const rememberRecentMedia = (openid, intent) => {
const publicUrl = String(intent?.media?.publicUrl ?? '').trim();
if (!publicUrl) return;
const key = String(openid ?? '').trim();
const now = Date.now();
const item = {
media: { ...(intent.media ?? {}) },
attachment: intent.attachment ? { ...intent.attachment } : null,
};
const current = recentMediaByOpenid.get(key);
const canAppendImage =
intent?.msgType === 'image' &&
current &&
!current.claimed &&
now - current.rememberedAt <= WECHAT_RECENT_MEDIA_TTL_MS &&
current.items.every((recentItem) => !recentItem.attachment);
const candidates = canAppendImage ? [...current.items, item] : [item];
const deduped = candidates.filter(
(candidate, index, values) =>
values.findIndex(
(value) => String(value?.media?.publicUrl ?? '') === String(candidate?.media?.publicUrl ?? ''),
) === index,
);
recentMediaByOpenid.set(key, {
items: deduped.slice(-WECHAT_RECENT_IMAGE_MAX_COUNT),
rememberedAt: now,
batchId: crypto.randomUUID(),
claimed: false,
});
};
const attachRecentMediaForFollowup = (openid, intent, mediaAnalysisEnabled) => {
if (!mediaAnalysisEnabled || intent?.msgType !== 'text' || intent?.media?.publicUrl) return;
const text = String(intent?.agentText ?? '');
if (!/(?:刚才|之前|上一|这张|这份|图片|图像|照片|文件|文档|表格|excel|word)/iu.test(text)) return;
const key = String(openid ?? '').trim();
const recent = recentMediaByOpenid.get(key);
if (
!recent ||
recent.claimed ||
Date.now() - recent.rememberedAt > WECHAT_RECENT_MEDIA_TTL_MS ||
recent.items.length === 0
) {
return;
}
const items = recent.items.map((item) => ({
media: { ...(item.media ?? {}), source: 'wechat_recent_media' },
attachment: item.attachment ? { ...item.attachment } : null,
}));
const primary = items.at(-1);
intent.media = { ...(primary?.media ?? {}) };
if (primary?.attachment) intent.attachment = { ...primary.attachment };
intent.recentMediaItems = items;
intent.recentMediaBatchId = recent.batchId;
recent.claimed = true;
};
const settleRecentMediaBatch = (openid, intent, { succeeded }) => {
const batchId = String(intent?.recentMediaBatchId ?? '').trim();
if (!batchId) return;
const key = String(openid ?? '').trim();
const recent = recentMediaByOpenid.get(key);
if (!recent || recent.batchId !== batchId) return;
if (succeeded) recentMediaByOpenid.delete(key);
else recent.claimed = false;
};
const resolveWechatBillingTokenState = (sessionId, tokenState) =>
resolveBillingTokenState(tokenState, {
sessionId,
@@ -1872,16 +2023,40 @@ export function createWechatMpService({
}
};
const buildIntentMetadata = (intent) => ({
source: 'wechat_mp',
msgType: intent.msgType,
originalMsgId: intent.msgId || null,
displayText: intent.displayText || '',
mediaPublicUrl: intent.media?.publicUrl || null,
recognition: intent.msgType === 'voice' ? intent.agentText || null : null,
location: intent.location || null,
link: intent.link || null,
});
const buildIntentMetadata = (intent, { mediaAnalysisEnabled = false } = {}) => {
const mediaPublicUrl = intent.media?.publicUrl || null;
const mediaItems = Array.isArray(intent.recentMediaItems) && intent.recentMediaItems.length > 0
? intent.recentMediaItems
: mediaPublicUrl
? [{ media: intent.media, attachment: intent.attachment ?? null }]
: [];
const imageUrls = mediaItems
.filter((item) => !item.attachment)
.map((item) => String(item?.media?.publicUrl ?? '').trim())
.filter((url, index, values) => url && values.indexOf(url) === index);
const fileAttachments = mediaItems
.filter((item) => item.attachment?.filename && item.media?.publicUrl)
.map((item) => ({
assetId: '',
downloadUrl: item.media.publicUrl,
filename: item.attachment.filename,
mimeType: item.attachment.mimeType || 'application/octet-stream',
}));
return {
source: 'wechat_mp',
msgType: intent.msgType,
originalMsgId: intent.msgId || null,
displayText: intent.displayText || '',
mediaPublicUrl,
...(mediaAnalysisEnabled && imageUrls.length > 0
? { imageUrls }
: {}),
...(mediaAnalysisEnabled && fileAttachments.length > 0 ? { fileAttachments } : {}),
recognition: intent.msgType === 'voice' ? intent.agentText || null : null,
location: intent.location || null,
link: intent.link || null,
};
};
const persistIntentDetail = async ({ intent, userId = null, rawXmlHash = '' }) => {
if (typeof userAuth.insertWechatMpMessageDetail !== 'function') return;
@@ -1921,6 +2096,7 @@ export function createWechatMpService({
const runIntentMessage = async ({ inbound, intent, user }) => {
const wechatIntent = classifyWechatIntent(intent);
const mediaAnalysisEnabled = isWechatMediaGrayUser(user, config.mediaAnalysisGrayUsers);
const resetCandidate =
intent.msgType === 'text' || intent.msgType === 'voice' ? intent.agentText : '';
// Page Data delivery owns persistent files, datasets and two publication
@@ -1970,7 +2146,18 @@ export function createWechatMpService({
sessionId,
requestId,
agentPrompt,
buildIntentMetadata(intent),
buildIntentMetadata(intent, { mediaAnalysisEnabled }),
{
submitReply: submitSessionReply
? ({ requestId: replyRequestId, userMessage }) =>
submitSessionReply({
userId: user.userId,
sessionId,
requestId: replyRequestId,
userMessage,
})
: null,
},
);
const workingDir = publishLayout?.publishDir ?? (await userAuth.resolveWorkingDir(user.userId));
const linkExistsForRequest = resolveLinkExistsForWorkingDir(workingDir, linkExists);
@@ -2129,7 +2316,18 @@ export function createWechatMpService({
sessionId,
retryId,
retryPrompt,
buildIntentMetadata(intent),
buildIntentMetadata(intent, { mediaAnalysisEnabled }),
{
submitReply: submitSessionReply
? ({ requestId: replyRequestId, userMessage }) =>
submitSessionReply({
userId: user.userId,
sessionId,
requestId: replyRequestId,
userMessage,
})
: null,
},
);
const workingDir = publishLayout?.publishDir ?? (await userAuth.resolveWorkingDir(user.userId));
const linkExistsForRequest = resolveLinkExistsForWorkingDir(workingDir, linkExists);
@@ -2302,6 +2500,7 @@ export function createWechatMpService({
const supportedByConfig =
(intent.msgType === 'voice' && config.acceptVoice) ||
(intent.msgType === 'image' && config.acceptImage) ||
(intent.msgType === 'file' && config.acceptFile) ||
(intent.msgType === 'location' && config.acceptLocation) ||
(intent.msgType === 'link' && config.acceptLink) ||
intent.msgType === 'text' ||
@@ -2350,6 +2549,26 @@ export function createWechatMpService({
};
}
const mediaAnalysisEnabled = isWechatMediaGrayUser(
boundUser,
config.mediaAnalysisGrayUsers,
);
attachRecentMediaForFollowup(inbound.fromUserName, intent, mediaAnalysisEnabled);
if (intent.msgType === 'file' && !mediaAnalysisEnabled) {
await persistIntentDetail({ intent, userId: boundUser.userId, rawXmlHash });
return {
ok: true,
status: 200,
contentType: 'application/xml; charset=utf-8',
body: buildWechatTextReply({
toUserName: inbound.fromUserName,
fromUserName: inbound.toUserName,
content: '当前账号尚未开启服务号文件分析灰度,请先通过 H5 上传文件。',
}),
};
}
if (intent.msgType === 'voice' && !intent.agentText.trim() && intent.media?.mediaId) {
try {
const fallbackText = await transcribeWechatVoiceMedia(intent.media.mediaId, intent.media.format);
@@ -2418,6 +2637,7 @@ export function createWechatMpService({
source: persisted.source,
};
intent.agentText = `[图片1]: ${persisted.publicUrl}`;
rememberRecentMedia(inbound.fromUserName, intent);
} catch (error) {
await persistIntentDetail({ intent, userId: boundUser.userId, rawXmlHash });
return {
@@ -2433,6 +2653,57 @@ export function createWechatMpService({
}
}
if (intent.msgType === 'file') {
try {
const accessToken = await getStableAccessToken();
const persisted = await persistWechatAttachment(
{
userId: boundUser.userId,
appId: config.appId,
openid: inbound.fromUserName,
msgId: inbound.msgId,
mediaId: inbound.mediaId,
filename: inbound.fileName,
publicBaseUrl: config.mediaPublicBaseUrl,
maxFileBytes: config.maxFileBytes,
},
{
wechatFetch,
accessToken,
},
);
intent.media = {
...intent.media,
mediaId: inbound.mediaId || intent.media?.mediaId || '',
publicUrl: persisted.publicUrl,
format: persisted.contentType,
source: persisted.source,
};
intent.attachment = {
...intent.attachment,
filename: persisted.filename,
publicUrl: persisted.publicUrl,
mimeType: persisted.contentType,
sizeBytes: persisted.bytes,
};
intent.agentText = `[文件1: ${persisted.filename}]: ${persisted.publicUrl}`;
intent.displayText = `文件:${persisted.filename}`;
rememberRecentMedia(inbound.fromUserName, intent);
} catch (error) {
await persistIntentDetail({ intent, userId: boundUser.userId, rawXmlHash });
return {
ok: true,
status: 200,
contentType: 'application/xml; charset=utf-8',
body: buildWechatTextReply({
toUserName: inbound.fromUserName,
fromUserName: inbound.toUserName,
content: error instanceof Error ? error.message : '文件处理失败,请稍后重试。',
}),
};
}
}
if (
intent.msgType === 'location' &&
(intent.location?.latitude == null || intent.location?.longitude == null)
@@ -2555,12 +2826,15 @@ export function createWechatMpService({
};
}
const task = runIntentMessage({
inbound,
intent,
user: boundUser,
})
const task = enqueueMessageTask(inbound.fromUserName, () =>
runIntentMessage({
inbound,
intent,
user: boundUser,
}),
)
.then(async ({ sessionId } = {}) => {
settleRecentMediaBatch(inbound.fromUserName, intent, { succeeded: true });
if (inbound.msgId && typeof userAuth.finishWechatMpMessage === 'function') {
await userAuth.finishWechatMpMessage({
appId: config.appId,
@@ -2572,6 +2846,7 @@ export function createWechatMpService({
}
})
.catch(async (err) => {
settleRecentMediaBatch(inbound.fromUserName, intent, { succeeded: false });
if (inbound.msgId && typeof userAuth.finishWechatMpMessage === 'function') {
await userAuth.finishWechatMpMessage({
appId: config.appId,
+554 -1
View File
@@ -79,6 +79,7 @@ function createBoundWechatService({
config = {},
scheduleService = null,
applySessionLlmProvider = null,
submitSessionReply = null,
}) {
return createWechatMpService({
config: {
@@ -128,6 +129,7 @@ function createBoundWechatService({
},
startAgentSession,
sessionApiFetch,
submitSessionReply,
scheduleService,
applySessionLlmProvider,
wechatFetch,
@@ -2712,6 +2714,10 @@ test('isRecoverableWechatAgentSessionError detects poisoned tool_calls history',
);
assert.equal(isRecoverableWechatAgentSessionError('stale_session_poisoned_completion'), true);
assert.equal(isRecoverableWechatAgentSessionError('无权访问该会话'), true);
assert.equal(
isRecoverableWechatAgentSessionError('Session already has an active request. Cancel it first.'),
true,
);
assert.equal(isRecoverableWechatAgentSessionError('network timeout'), false);
});
@@ -2752,6 +2758,17 @@ test('sanitizeWechatAgentOutboundText replaces raw api errors with friendly text
);
});
test('assertWechatAgentReplyIsSendable rejects image inspection placeholders', () => {
assert.throws(
() => assertWechatAgentReplyIsSendable({ text: 'Let me look at the image first.' }),
/wechat_agent_incomplete_reply/,
);
assert.throws(
() => assertWechatAgentReplyIsSendable({ text: '我先看一下这张图片。' }),
/wechat_agent_incomplete_reply/,
);
});
test('wechat mp service recreates dedicated session when tool_calls error arrives via Finish assistant text', async () => {
const token = 'token';
const timestamp = '1710000000';
@@ -3680,12 +3697,13 @@ test('wechat mp service falls back to ASR when voice recognition is empty', asyn
}
});
test('wechat mp service persists image and routes image url into agent prompt', async () => {
test('wechat mp wildcard media access persists image and routes image url into agent prompt', async () => {
const token = 'token';
const timestamp = '1710000000';
const nonce = 'nonce';
const testUserId = 'test-user-image';
const prompts = [];
const metadataCalls = [];
const detailCalls = [];
const service = createWechatMpService({
config: {
@@ -3701,6 +3719,7 @@ test('wechat mp service persists image and routes image url into agent prompt',
unboundTextPrefix: '请先绑定',
progressDelayMs: 0,
maxImageBytes: 1024 * 1024,
mediaAnalysisGrayUsers: ['*'],
},
userAuth: {
async findWechatUserByOpenid() {
@@ -3747,6 +3766,7 @@ test('wechat mp service persists image and routes image url into agent prompt',
if (pathname === '/sessions/session-1/reply') {
const body = JSON.parse(init.body);
prompts.push(body.user_message.content[0].text);
metadataCalls.push(body.user_message.metadata);
return new Response('{}', { status: 200, headers: { 'Content-Type': 'application/json' } });
}
if (pathname === '/agent/harness_remember' || pathname === '/agent/harness_bootstrap') {
@@ -3809,10 +3829,543 @@ test('wechat mp service persists image and routes image url into agent prompt',
prompts[0],
/\[图片1\]: https:\/\/example\.com\/MindSpace\/test-user-image\/public\/wechat-mp\//,
);
assert.equal(metadataCalls.length, 1);
assert.equal(metadataCalls[0].source, 'wechat_mp');
assert.equal(metadataCalls[0].msgType, 'image');
assert.equal(metadataCalls[0].imageUrls.length, 1);
assert.match(metadataCalls[0].imageUrls[0], /\/public\/wechat-mp\//);
assert.equal(detailCalls.length, 1);
assert.match(detailCalls[0].mediaPublicUrl, /\/wechat-mp\//);
});
test('wechat mp image submission reuses the H5 prepared reply path', async () => {
const token = 'token';
const timestamp = '1710000000';
const nonce = 'nonce';
const testUserId = 'test-user-image-prepared';
const submitCalls = [];
const service = createBoundWechatService({
token,
config: {
mediaAnalysisGrayUsers: [testUserId],
},
userAuth: {
async findWechatUserByOpenid() {
return { userId: testUserId, status: 'active', nickname: '唐' };
},
},
sessionApiFetch: async (_sessionId, pathname) => {
if (pathname === '/sessions/session-1/events') {
return new Response(
[
'data: {"type":"Message","message":{"id":"assistant-1","role":"assistant","metadata":{"userVisible":true},"content":[{"type":"text","text":"图片内容已识别。"}]}}\n\n',
'data: {"type":"Finish"}\n\n',
].join(''),
{ status: 200, headers: { 'Content-Type': 'text/event-stream' } },
);
}
if (pathname === '/agent/harness_remember' || pathname === '/agent/harness_bootstrap') {
return new Response('{}', { status: 200, headers: { 'Content-Type': 'application/json' } });
}
throw new Error(`unexpected api path: ${pathname}`);
},
submitSessionReply: async (input) => {
submitCalls.push(input);
return { ok: true };
},
wechatFetch: async (url) => {
if (String(url).includes('/cgi-bin/stable_token')) {
return new Response(JSON.stringify({ access_token: 'access-1', expires_in: 7200 }), {
status: 200,
headers: { 'Content-Type': 'application/json' },
});
}
if (String(url).includes('/cgi-bin/media/get')) {
return new Response(Buffer.from([0x89, 0x50, 0x4e, 0x47]), {
status: 200,
headers: { 'Content-Type': 'image/png' },
});
}
if (String(url).includes('/cgi-bin/message/custom/send')) {
return new Response(JSON.stringify({ errcode: 0, errmsg: 'ok' }), {
status: 200,
headers: { 'Content-Type': 'application/json' },
});
}
throw new Error(`unexpected wechat url: ${url}`);
},
});
try {
const result = await service.handleInboundMessage(
inboundXml({
msgType: 'image',
content: '',
extraFields: { MediaId: 'media-prepared', PicUrl: 'https://wx.example.com/image.png' },
}),
{ timestamp, nonce, signature: signatureFor(token, timestamp, nonce) },
);
await result.task;
} finally {
fs.rmSync(path.join(process.cwd(), 'MindSpace', testUserId), { recursive: true, force: true });
}
assert.equal(submitCalls.length, 1);
assert.equal(submitCalls[0].userId, testUserId);
assert.equal(submitCalls[0].sessionId, 'session-1');
assert.equal(submitCalls[0].userMessage.metadata.imageUrls.length, 1);
assert.match(submitCalls[0].userMessage.metadata.imageUrls[0], /\/public\/wechat-mp\//);
});
test('wechat mp serializes image and follow-up text and reattaches recent image', async () => {
const token = 'token';
const timestamp = '1710000000';
const nonce = 'nonce';
const testUserId = 'test-user-image-followup';
const submitCalls = [];
let eventCall = 0;
let releaseFirst = null;
const service = createBoundWechatService({
token,
config: {
mediaAnalysisGrayUsers: [testUserId],
},
userAuth: {
async findWechatUserByOpenid() {
return { userId: testUserId, status: 'active', nickname: '唐' };
},
},
sessionApiFetch: async (_sessionId, pathname) => {
if (pathname === '/sessions/session-1/events') {
eventCall += 1;
if (eventCall === 1) {
return new Response(
new ReadableStream({
start(controller) {
releaseFirst = () => {
controller.enqueue(
new TextEncoder().encode(
'data: {"type":"Message","message":{"id":"assistant-image","role":"assistant","metadata":{"userVisible":true},"content":[{"type":"text","text":"图片已识别。"}]}}\n\n' +
'data: {"type":"Finish"}\n\n',
),
);
controller.close();
};
},
}),
{ status: 200, headers: { 'Content-Type': 'text/event-stream' } },
);
}
return new Response(
[
'data: {"type":"Message","message":{"id":"assistant-followup","role":"assistant","metadata":{"userVisible":true},"content":[{"type":"text","text":"已结合刚才图片分析主题。"}]}}\n\n',
'data: {"type":"Finish"}\n\n',
].join(''),
{ status: 200, headers: { 'Content-Type': 'text/event-stream' } },
);
}
if (pathname === '/agent/harness_remember' || pathname === '/agent/harness_bootstrap') {
return new Response('{}', { status: 200, headers: { 'Content-Type': 'application/json' } });
}
throw new Error(`unexpected api path: ${pathname}`);
},
submitSessionReply: async (input) => {
submitCalls.push(input);
return { ok: true };
},
wechatFetch: async (url) => {
if (String(url).includes('/cgi-bin/stable_token')) {
return new Response(JSON.stringify({ access_token: 'access-1', expires_in: 7200 }), {
status: 200,
headers: { 'Content-Type': 'application/json' },
});
}
if (String(url).includes('/cgi-bin/media/get')) {
return new Response(Buffer.from([0x89, 0x50, 0x4e, 0x47]), {
status: 200,
headers: { 'Content-Type': 'image/png' },
});
}
if (String(url).includes('/cgi-bin/message/custom/send')) {
return new Response(JSON.stringify({ errcode: 0, errmsg: 'ok' }), {
status: 200,
headers: { 'Content-Type': 'application/json' },
});
}
throw new Error(`unexpected wechat url: ${url}`);
},
});
try {
const imageResult = await service.handleInboundMessage(
inboundXml({
msgType: 'image',
content: '',
extraFields: { MediaId: 'media-followup', PicUrl: 'https://wx.example.com/image.png' },
}),
{ timestamp, nonce, signature: signatureFor(token, timestamp, nonce) },
);
while (submitCalls.length === 0) await new Promise((resolve) => setTimeout(resolve, 0));
const followupResult = await service.handleInboundMessage(
inboundXml({ msgType: 'text', content: '请根据刚才图片分析主题' }),
{ timestamp, nonce, signature: signatureFor(token, timestamp, nonce) },
);
await new Promise((resolve) => setTimeout(resolve, 0));
assert.equal(submitCalls.length, 1);
releaseFirst();
await imageResult.task;
await followupResult.task;
} finally {
fs.rmSync(path.join(process.cwd(), 'MindSpace', testUserId), { recursive: true, force: true });
}
assert.equal(submitCalls.length, 2);
assert.deepEqual(
submitCalls[1].userMessage.metadata.imageUrls,
submitCalls[0].userMessage.metadata.imageUrls,
);
assert.equal(submitCalls[1].userMessage.metadata.msgType, 'text');
});
test('wechat mp groups consecutive images for one follow-up and clears the consumed batch', async () => {
const token = 'token';
const timestamp = '1710000000';
const nonce = 'nonce';
const testUserId = 'test-user-multi-image-followup';
const submitCalls = [];
let eventCall = 0;
let releaseFirst = null;
const service = createBoundWechatService({
token,
config: {
mediaAnalysisGrayUsers: [testUserId],
},
userAuth: {
async findWechatUserByOpenid() {
return { userId: testUserId, status: 'active', nickname: '唐' };
},
},
sessionApiFetch: async (_sessionId, pathname) => {
if (pathname === '/sessions/session-1/events') {
eventCall += 1;
if (eventCall === 1) {
return new Response(
new ReadableStream({
start(controller) {
releaseFirst = () => {
controller.enqueue(
new TextEncoder().encode(
'data: {"type":"Message","message":{"id":"assistant-first","role":"assistant","metadata":{"userVisible":true},"content":[{"type":"text","text":"第一张处理完成。"}]}}\n\n' +
'data: {"type":"Finish"}\n\n',
),
);
controller.close();
};
},
}),
{ status: 200, headers: { 'Content-Type': 'text/event-stream' } },
);
}
return new Response(
[
'data: {"type":"Message","message":{"id":"assistant-ok","role":"assistant","metadata":{"userVisible":true},"content":[{"type":"text","text":"处理完成。"}]}}\n\n',
'data: {"type":"Finish"}\n\n',
].join(''),
{ status: 200, headers: { 'Content-Type': 'text/event-stream' } },
);
}
if (pathname === '/agent/harness_remember' || pathname === '/agent/harness_bootstrap') {
return new Response('{}', { status: 200, headers: { 'Content-Type': 'application/json' } });
}
throw new Error(`unexpected api path: ${pathname}`);
},
submitSessionReply: async (input) => {
submitCalls.push(input);
return { ok: true };
},
wechatFetch: async (url) => {
if (String(url).includes('/cgi-bin/stable_token')) {
return new Response(JSON.stringify({ access_token: 'access-1', expires_in: 7200 }), {
status: 200,
headers: { 'Content-Type': 'application/json' },
});
}
if (String(url).includes('/cgi-bin/media/get')) {
return new Response(Buffer.from([0x89, 0x50, 0x4e, 0x47]), {
status: 200,
headers: { 'Content-Type': 'image/png' },
});
}
if (String(url).includes('/cgi-bin/message/custom/send')) {
return new Response(JSON.stringify({ errcode: 0, errmsg: 'ok' }), {
status: 200,
headers: { 'Content-Type': 'application/json' },
});
}
throw new Error(`unexpected wechat url: ${url}`);
},
});
try {
const firstImageResult = await service.handleInboundMessage(
inboundXml({
msgType: 'image',
content: '',
extraFields: {
MsgId: '10011',
MediaId: 'media-first',
PicUrl: 'https://wx.example.com/media-first.png',
},
}),
{ timestamp, nonce, signature: signatureFor(token, timestamp, nonce) },
);
while (submitCalls.length === 0) await new Promise((resolve) => setTimeout(resolve, 0));
const secondImageResult = await service.handleInboundMessage(
inboundXml({
msgType: 'image',
content: '',
extraFields: {
MsgId: '10012',
MediaId: 'media-second',
PicUrl: 'https://wx.example.com/media-second.png',
},
}),
{ timestamp, nonce, signature: signatureFor(token, timestamp, nonce) },
);
const followupResult = await service.handleInboundMessage(
inboundXml({
content: '请结合刚才两张图片分析主题',
extraFields: { MsgId: '10013' },
}),
{ timestamp, nonce, signature: signatureFor(token, timestamp, nonce) },
);
await new Promise((resolve) => setTimeout(resolve, 0));
assert.equal(submitCalls.length, 1);
releaseFirst();
await firstImageResult.task;
await secondImageResult.task;
await followupResult.task;
const laterResult = await service.handleInboundMessage(
inboundXml({
content: '请再次分析刚才图片',
extraFields: { MsgId: '10014' },
}),
{ timestamp, nonce, signature: signatureFor(token, timestamp, nonce) },
);
await laterResult.task;
} finally {
fs.rmSync(path.join(process.cwd(), 'MindSpace', testUserId), { recursive: true, force: true });
}
assert.equal(submitCalls.length, 4);
assert.equal(submitCalls[0].userMessage.metadata.imageUrls.length, 1);
assert.equal(submitCalls[1].userMessage.metadata.imageUrls.length, 1);
assert.equal(submitCalls[2].userMessage.metadata.imageUrls.length, 2);
assert.match(submitCalls[2].userMessage.metadata.imageUrls[0], /media-first/);
assert.match(submitCalls[2].userMessage.metadata.imageUrls[1], /media-second/);
assert.equal(submitCalls[2].userMessage.metadata.msgType, 'text');
assert.equal(submitCalls[3].userMessage.metadata.imageUrls, undefined);
});
test('wechat mp service persists Word and Excel files in user public area and reuses H5 attachment metadata', async () => {
const token = 'token';
const timestamp = '1710000000';
const nonce = 'nonce';
const testUserId = 'test-user-wechat-file';
const userMessages = [];
const service = createBoundWechatService({
token,
config: {
maxFileBytes: 1024 * 1024,
acceptFile: true,
mediaAnalysisGrayUsers: [testUserId],
},
userAuth: {
async findWechatUserByOpenid() {
return { userId: testUserId, status: 'active', nickname: '毕升' };
},
},
sessionApiFetch: async (_sessionId, pathname, init = {}) => {
if (pathname === '/sessions/session-1/events') {
return new Response(
[
'data: {"type":"Message","request_id":"req-file","message":{"id":"assistant-1","role":"assistant","metadata":{"userVisible":true},"content":[{"type":"text","text":"文件已解析。"}]}}\n\n',
'data: {"type":"Finish","request_id":"req-file","token_state":{"inputTokens":1,"outputTokens":2}}\n\n',
].join(''),
{ status: 200, headers: { 'Content-Type': 'text/event-stream' } },
);
}
if (pathname === '/sessions/session-1/reply') {
const body = JSON.parse(init.body);
userMessages.push(body.user_message);
return new Response('{}', { status: 200, headers: { 'Content-Type': 'application/json' } });
}
if (pathname === '/agent/harness_remember' || pathname === '/agent/harness_bootstrap') {
return new Response('{}', { status: 200, headers: { 'Content-Type': 'application/json' } });
}
throw new Error(`unexpected api path: ${pathname}`);
},
wechatFetch: async (url) => {
if (String(url).includes('/cgi-bin/stable_token')) {
return new Response(JSON.stringify({ access_token: 'access-1', expires_in: 7200 }), {
status: 200,
headers: { 'Content-Type': 'application/json' },
});
}
if (String(url).includes('/cgi-bin/media/get')) {
return new Response(Buffer.from('fake-docx-content'), {
status: 200,
headers: { 'Content-Type': 'application/octet-stream' },
});
}
if (String(url).includes('/cgi-bin/message/custom/send')) {
return new Response(JSON.stringify({ errcode: 0, errmsg: 'ok' }), {
status: 200,
headers: { 'Content-Type': 'application/json' },
});
}
throw new Error(`unexpected wechat url: ${url}`);
},
});
const originalRandomUuid = crypto.randomUUID;
crypto.randomUUID = () => 'req-file';
try {
for (const [index, filename] of ['季度分析.docx', '销售数据.xlsx'].entries()) {
const result = await service.handleInboundMessage(
inboundXml({
msgType: 'file',
content: '',
extraFields: {
MediaId: `file-media-${index + 1}`,
FileName: filename,
FileSize: 17,
},
}),
{ timestamp, nonce, signature: signatureFor(token, timestamp, nonce) },
);
assert.equal(result.status, 200);
await result.task;
const attachment = userMessages.at(-1)?.metadata?.fileAttachments?.[0];
const publicFilename = decodeURIComponent(new URL(attachment.downloadUrl).pathname.split('/').at(-1));
assert.equal(
fs.existsSync(
path.join(process.cwd(), 'MindSpace', testUserId, 'public', 'wechat-mp', publicFilename),
),
true,
);
}
} finally {
crypto.randomUUID = originalRandomUuid;
fs.rmSync(path.join(process.cwd(), 'MindSpace', testUserId), {
recursive: true,
force: true,
});
}
assert.equal(userMessages.length, 2);
assert.match(userMessages[0].content[0].text, /【微信服务号文件消息】/);
assert.match(userMessages[0].content[0].text, /\[文件1: 季度分析\.docx\]:/);
assert.equal(userMessages[0].metadata.msgType, 'file');
assert.equal(userMessages[0].metadata.fileAttachments.length, 1);
assert.equal(userMessages[0].metadata.fileAttachments[0].filename, '季度分析.docx');
assert.equal(
userMessages[0].metadata.fileAttachments[0].mimeType,
'application/vnd.openxmlformats-officedocument.wordprocessingml.document',
);
assert.match(userMessages[0].metadata.fileAttachments[0].downloadUrl, /\/public\/wechat-mp\//);
assert.match(userMessages[1].content[0].text, /\[文件1: 销售数据\.xlsx\]:/);
assert.equal(userMessages[1].metadata.fileAttachments[0].filename, '销售数据.xlsx');
assert.equal(
userMessages[1].metadata.fileAttachments[0].mimeType,
'application/vnd.openxmlformats-officedocument.spreadsheetml.sheet',
);
assert.match(userMessages[1].metadata.fileAttachments[0].downloadUrl, /\/public\/wechat-mp\//);
});
test('wechat mp service rejects unsupported public file types without entering the agent flow', async () => {
let mediaDownloads = 0;
let sessionCalls = 0;
const service = createBoundWechatService({
config: { acceptFile: true, mediaAnalysisGrayUsers: ['user-1'] },
sessionApiFetch: async () => {
sessionCalls += 1;
throw new Error('session should not be called');
},
wechatFetch: async (url) => {
if (String(url).includes('/cgi-bin/stable_token')) {
return new Response(JSON.stringify({ access_token: 'access-1', expires_in: 7200 }), {
status: 200,
headers: { 'Content-Type': 'application/json' },
});
}
if (String(url).includes('/cgi-bin/media/get')) mediaDownloads += 1;
throw new Error(`unexpected wechat url: ${url}`);
},
});
const result = await service.handleInboundMessage(
inboundXml({
msgType: 'file',
content: '',
extraFields: { MediaId: 'file-media-2', FileName: 'payload.exe' },
}),
{
timestamp: '1710000000',
nonce: 'nonce',
signature: signatureFor('token', '1710000000', 'nonce'),
},
);
assert.equal(result.status, 200);
assert.match(result.body, /仅支持 Word/);
assert.equal(mediaDownloads, 0);
assert.equal(sessionCalls, 0);
});
test('wechat mp service keeps file analysis disabled outside the media gray allowlist', async () => {
let mediaDownloads = 0;
let sessionCalls = 0;
const service = createBoundWechatService({
config: {
acceptFile: true,
mediaAnalysisGrayUsers: ['唐'],
},
sessionApiFetch: async () => {
sessionCalls += 1;
throw new Error('session should not be called');
},
wechatFetch: async (url) => {
if (String(url).includes('/cgi-bin/media/get')) mediaDownloads += 1;
throw new Error(`unexpected wechat url: ${url}`);
},
});
const result = await service.handleInboundMessage(
inboundXml({
msgType: 'file',
content: '',
extraFields: { MediaId: 'file-media-gray', FileName: '测试.docx' },
}),
{
timestamp: '1710000000',
nonce: 'nonce',
signature: signatureFor('token', '1710000000', 'nonce'),
},
);
assert.equal(result.status, 200);
assert.match(result.body, /尚未开启服务号文件分析灰度/);
assert.equal(mediaDownloads, 0);
assert.equal(sessionCalls, 0);
});
test('wechat mp service accepts full voice xml payload and routes recognition text into agent', async () => {
const token = 'token';
const timestamp = '1710000000';
+7 -1
View File
@@ -21,6 +21,12 @@ const ImageBuilder = {
build: (ctx) => buildText(TEMPLATES.image, ctx),
};
const FileBuilder = {
priority: 80,
support: (ctx) => ctx.msgType === 'file',
build: (ctx) => buildText(TEMPLATES.file, ctx),
};
const VoiceBuilder = {
priority: 80,
support: (ctx) => ctx.msgType === 'voice',
@@ -54,7 +60,7 @@ const DefaultBuilder = {
build: (ctx) => buildText(TEMPLATES.default, ctx),
};
const BUILDERS = [ImageBuilder, VoiceBuilder, LocationBuilder, LinkBuilder, IntentBuilder, DefaultBuilder]
const BUILDERS = [ImageBuilder, FileBuilder, VoiceBuilder, LocationBuilder, LinkBuilder, IntentBuilder, DefaultBuilder]
.sort((a, b) => b.priority - a.priority);
export function selectBuilder(ctx) {
+1
View File
@@ -13,6 +13,7 @@ const INTENT_RULES = [
export function resolveIntent(msgType, text) {
if (msgType === 'image') return { task: 'image_analysis' };
if (msgType === 'file') return { task: 'file_analysis' };
if (msgType === 'voice') return { task: 'voice_analysis' };
if (msgType === 'location') return { task: 'location' };
if (msgType === 'link') return { task: 'link' };
+5
View File
@@ -26,6 +26,11 @@ describe('buildAckText', () => {
assert.equal(result, '图片收到,我先看看。');
});
it('file → file template', () => {
const result = buildAckText({ intent: intent('file'), nickname: '', config: cfg, fallbackText: '' });
assert.equal(result, '文件收到,我先读取分析。');
});
it('voice → voice template', () => {
const result = buildAckText({ intent: intent('voice'), nickname: '', config: cfg, fallbackText: '' });
assert.equal(result, '收到语音,我先听一下。');
+1
View File
@@ -12,6 +12,7 @@ export const TEMPLATES = {
'让我看看这张图片。',
'图片已收到,我来处理。',
],
file: ['文件收到,我先读取分析。'],
voice: [
'收到语音,我先听一下。',
'听到啦,我马上处理。',
+14
View File
@@ -81,6 +81,20 @@ export function buildWechatAgentPrompt(intent, { grantedSkills = [] } = {}) {
.filter(Boolean)
.join('\n');
}
if (msgType === 'file') {
const attachment = intent?.attachment ?? {};
return [
currentTimeHint,
'【微信服务号文件消息】用户发送了需要解析的 Office 文件。',
attachment.filename ? `文件名:${attachment.filename}` : '',
attachment.publicUrl ? `文件链接:${attachment.publicUrl}` : '',
'请复用 H5 附件分析结果回答;Excel 在专用工具可用时必须优先使用 Excel 工具读取完整工作簿。',
'',
String(agentText).trim(),
]
.filter(Boolean)
.join('\n');
}
if (msgType === 'location') {
const location = intent?.location ?? {};
return [