diff --git a/docs/architecture/memfuse-bench-baseline.md b/docs/architecture/memfuse-bench-baseline.md new file mode 100644 index 0000000..c4a6008 --- /dev/null +++ b/docs/architecture/memfuse-bench-baseline.md @@ -0,0 +1,132 @@ +# MemFuseBench 检索基线(Memory V2) + +日期: 2026-09-02 + +状态: 已落地测量工具 + 首份基线。**不改动任何生产运行时路径**,纯离线评测。 + +## 0. 结论 + +在 MemFuseBench 的 357 个证据锚定问题、7823 条来源标注事件上跑现网 pgvector 检索排序链路,得到: + +| 指标 | k=5 | k=20 | k=50 | +|---|---|---|---| +| 候选召回 candidateRecall | 36.6% | 36.6% | 36.6% | +| 最终召回 recall@k | 6.8% | 13.5% | 22.5% | +| **排序丢失 rankingLoss** | **29.8%** | **23.1%** | **14.2%** | +| 至少命中一条 hitAny@k | 21.8% | 41.7% | 64.4% | +| 清单覆盖 checklistCoverage | 7.8% | 16.8% | 28.3% | +| MRR | 0.135 | 0.153 | 0.160 | +| 干扰项占比 distractorRate | 8.5% | 10.8% | 12.6% | + +两个可操作结论: + +1. **天花板在候选生成,不在 k。** candidateRecall 恒为 36.6%,与 k 无关——超过 63% 的证据从未进入候选池,再怎么调排序或放大 k 都拿不回来。 +2. **进了候选池的证据,排序又丢掉了一大半。** k=20 时候选池已包含 36.6% 的证据,最终只留下 13.5%,rankingLoss 23.1 个百分点。即使 k=50(在约 200 条候选里返回 50 条),仍丢 14.2 个百分点。MRR 从 k=5 到 k=50 只从 0.135 涨到 0.160,说明列表头部的质量与 k 无关地差。 + +按维度看,`multi_source_conflict_arbitration` 明显最好(候选 71.5% / 召回 39.4%),`cross_device_information_fusion` 最差(26.5% / 6.1%)。 + +## 1. 两个已定位的机制 + +### 机制一:`lexicalQueryCoverage` 是主排序键,但判别力接近随机 + +`memory-v2-pgvector.mjs` 的 `rankHybridCandidates` 把 `lexicalScore` 作为**第一排序键**,向量分只在 lexical 分相同时才起作用。而 `lexicalQueryCoverage` 是字符 2-gram 覆盖率。全量 357 题实测: + +- gold 事件平均 coverage:`0.5366` +- 非 gold 平均 coverage:`0.4853` +- gold 高于语料中位数的比例:`64.5%`(随机为 50%) + +也就是一个只比随机好 14.5 个百分点的弱信号,被用作压倒性的主排序键。在英文语料上排名最高的往往是最长、bigram 最密的无关句子。 + +这个启发式对中文是合理的——CJK 2-gram 近似于词,判别力强;对英文退化严重,因为 `th`/`he`/`in` 这类 bigram 在任何长句里都存在。 + +### 机制二:keyword 回退路径按时间排序截断,不按相关性 + +`fetchKeywordCandidates` 的 SQL 是 `... WHERE content ILIKE ANY(...) ORDER BY updated_at DESC LIMIT n`。全量实测: + +- ILIKE 能匹配到的 gold:`60.4%` +- 按 `updated_at DESC` 截断 100 条后存活的 gold:`14.2%` +- 至少存活一条 gold 的问题占比:`32.5%` + +一次纯粹由"按时间截断"造成的 46 个百分点损失。语料里匹配关键词的事件一旦多于 limit,这条路召回的就是"最新的匹配"而不是"最相关的匹配"。 + +## 2. 必须一起读的三条限制 + +不要把上面的绝对数字当成生产召回率。 + +1. **默认嵌入是 `lexical-hash`,不是语义嵌入。** 它是哈希词袋 + L2 归一化的确定性离线替身,cosine 近似词面重叠。报告里 `embeddingMode` 会标出来。机制一和机制二与嵌入无关(都在 lexical / keyword 路径上),但 36.6% 这个候选天花板里有一部分要归因于替身嵌入弱。用 `--embedding-module` 接真实嵌入后才能给向量路径定论。 +2. **语料是英文,而 lexical 层是按中文调的。** 见机制一。这既是限制也是发现:一旦 Memind 要处理非中文内容,这条排序键就失效。 +3. **语料粒度比现网粗。** MemFuseBench 的语料是事件级(单场景 936–1886 条原始事件),而现网 `memory_embeddings` 存的是已经筛过的用户记忆,条数少得多。所以这份基线衡量的是"如果把原始事件流直接灌进现有检索层会怎样"——正好是 Event Kernel 方向会产生的形态。 + +## 3. 数据集与授权 + +**数据集不入库。** 它在仓库外,通过 `MEMFUSE_BENCH_DATASET` 或 `--dataset` 指定;缺失时所有入口返回 `{ available: false, reason }`,CLI 打印 skip 并 exit 0(`--strict` 才 exit 1)。CI 无需数据集即可全绿。 + +```bash +git clone https://github.com/Darwin-Agent/Mi-Memory ~/Project/mi-memory +``` + +默认路径 `~/Project/mi-memory/MemFuse/MemFuseBench/memfusebench_dataset.json`(6.2MB)。 + +| 项目 | 授权 | 本仓库可以做什么 | +|---|---|---| +| Mi-Memory / MemFuseBench | MIT(Darwin Agent Team, Xiaomi) | 可自由使用数据集与论文思想 | +| JKRiver / Riverse | **AGPL-3.0 或商业双授权** | **只做架构级参考,禁止移植代码** | + +JKRiver 的 `LICENSE` 明确把"集成进专有产品"和"作为商业 SaaS 提供而不开源修改"列为需要单独商业授权的场景。Memind 是带计费的商业 SaaS,AGPL 的网络 copyleft 正好命中。因此: + +- 允许:借鉴设计决策(如整条流水线单事务 + 末步落 processed 水位的 at-least-once 幂等)、参考反面经验(`migrations/008_drop_hypotheses.sql` 记录了 hypotheses 表建后即废,被 `layer='suspected'/'confirmed'` + supersedes 取代)。 +- 禁止:把 `agent/sleep/*.py`(含 `_maturity.py` 的分层衰减表、`disputes.py` 的争议解决)逐行翻译进 `memory-v2-lifecycle.mjs`。取得商业授权前不做。 + +Mi-Memory 仓库本身**没有代码**,只有论文 PDF、项目页和图;MemStack / MemSense / D²ACCI / E²MEND / LiteMem 均无参考实现。唯一可直接使用的产物就是 MemFuseBench 数据集。 + +## 4. 用法 + +```bash +# 全量 357 题,约 3.5s +npm run bench:memory-v2-memfuse + +# 单场景 / 单维度 / 冒烟 +node scripts/run-memory-v2-memfuse-bench.mjs --scenario sc1 --limit 20 +node scripts/run-memory-v2-memfuse-bench.mjs --dimension multi_source_conflict_arbitration +node scripts/run-memory-v2-memfuse-bench.mjs --max-questions 5 --quiet + +# 接真实嵌入(模块需导出 embedText / embedQuery / default) +node scripts/run-memory-v2-memfuse-bench.mjs --embedding-module ./scripts/my-embedder.mjs + +# 事件文本前置 "[device · location]" 来源标签 +node scripts/run-memory-v2-memfuse-bench.mjs --source-tags + +# 落 JSON 报告 +node scripts/run-memory-v2-memfuse-bench.mjs --out .release-gate/memfuse.json +``` + +`--source-tags` 实测把候选召回从 36.6% 抬到 38.2%,但最终召回不变(13.5%),说明当前排序器吃不到 provenance 前缀带来的信息。 + +单测(不依赖外部数据集): + +```bash +npm run verify:memory-v2-memfuse-bench +``` + +## 5. 指标定义 + +- `candidateRecall` — 证据事件进入候选池(向量 top-N ∪ 最近 N ∪ keyword 匹配)的比例。**这是排序无关的天花板。** +- `recall@k` — 证据事件出现在最终 top-k 的比例。 +- `rankingLoss` = `candidateRecall - recall@k`。大于 0 表示证据已被检索到但被排序丢弃。**这是决定"该修检索还是该修排序"的关键拆分**,对应 Mi-Memory 讲的 diagnostic trace。 +- `hitAny@k` — 至少召回一条证据的问题占比。 +- `checklistCoverage` — `answer_checklist` 中每个 point 至少召回一条其 `source_events` 的比例。这是最接近 MemFuse 论文自身评分(answer-checklist coverage)的检索侧代理,无需 LLM 评审。 +- `distractorRate` — top-k 中 `source` 为 `noise` / `adversarial` 的占比。这些事件按数据集构造永远不属于任何答案的证据集。 + +## 6. 这份基线解锁了什么 + +`memory-v2-lifecycle.mjs` 的 `promote` / `compact` / `reflect` / decay 目前 flag 全关、`reflect()` 恒返回 `updated: 0`。在没有基线之前,往里面填任何算法都无法验证对错。现在有了一个 3.5 秒可重跑、357 题、能把候选损失和排序损失分开的度量,这个前置条件解除。 + +按 rankingLoss 的量级,优先级排序是:先修排序(`rankHybridCandidates` 的主键选择、`fetchKeywordCandidates` 的排序方式),再谈候选生成和 consolidation 算法。这两处都是几十行的局部改动,而且每一步都能用本文的表格直接对比。 + +## 7. 相关文件 + +- `memory-v2-memfuse-bench.mjs` — 数据集加载、语料/案例构建、评测与聚合 +- `memory-v2-memfuse-bench.test.mjs` — 内联 fixture,19 个用例,不依赖外部数据集 +- `scripts/run-memory-v2-memfuse-bench.mjs` — CLI +- `memory-v2-pgvector.mjs` — 被测的生产排序路径(`rankHybridCandidates` / `fetchKeywordCandidates` / `extractKeywordTerms`) +- `memory-v2-recall-benchmark.mjs` — 既有的 4 例中文手写基线,与本文互补 diff --git a/memory-v2-memfuse-bench.mjs b/memory-v2-memfuse-bench.mjs new file mode 100644 index 0000000..29039c8 --- /dev/null +++ b/memory-v2-memfuse-bench.mjs @@ -0,0 +1,600 @@ +import os from 'node:os'; +import path from 'node:path'; +import { readFile as fsReadFile } from 'node:fs/promises'; + +import { createPgvectorMemoryBackend } from './memory-v2-pgvector.mjs'; + +/** + * MemFuseBench retrieval harness for Memory V2. + * + * MemFuseBench (Mi-Memory / Darwin Agent Team, MIT) ships 357 evidence-grounded + * questions over 7,823 source-tagged events. This module turns those scenarios + * into a corpus + case list and runs them through the *production* pgvector + * backend so the measured numbers reflect `rankHybridCandidates` and + * `extractKeywordTerms`, not a parallel scoring implementation. + * + * The dataset itself is never vendored into this repository. It is read from an + * external checkout and every entry point degrades to + * `{ available: false, reason }` when the file is absent, so CI stays green + * without it. + */ + +export const MEMFUSE_DATASET_ENV = 'MEMFUSE_BENCH_DATASET'; + +export const MEMFUSE_DIMENSIONS = Object.freeze([ + 'cross_device_causal_reasoning', + 'cross_device_information_fusion', + 'multi_source_conflict_arbitration', + 'cross_user_information_synthesis', + 'cross_user_query', + 'perspective_difference', +]); + +// Events tagged with these sources are adversarial by construction: they look +// topical but are never part of any answer's evidence set. +export const MEMFUSE_DISTRACTOR_SOURCES = Object.freeze(['noise', 'adversarial']); + +const DEFAULT_RELATIVE_DATASET_PATH = path.join( + 'mi-memory', + 'MemFuse', + 'MemFuseBench', + 'memfusebench_dataset.json', +); + +export function resolveMemFuseDatasetPath(env = process.env) { + const explicit = String(env?.[MEMFUSE_DATASET_ENV] ?? '').trim(); + if (explicit) return path.resolve(explicit); + const projectRoot = String(env?.MEMFUSE_PROJECT_ROOT ?? '').trim(); + const base = projectRoot || path.join(os.homedir(), 'Project'); + return path.join(base, DEFAULT_RELATIVE_DATASET_PATH); +} + +function asArray(value) { + return Array.isArray(value) ? value : []; +} + +function nonEmptyString(value) { + const text = String(value ?? '').trim(); + return text.length > 0 ? text : null; +} + +export function normalizeMemFuseDataset(raw) { + const scenarios = asArray(raw?.scenarios) + .map((scenario) => { + const scenarioId = nonEmptyString(scenario?.scenario_id); + if (!scenarioId) return null; + const episodes = asArray(scenario?.episodes) + .map((episode) => { + const episodeId = nonEmptyString(episode?.episode_id); + const events = asArray(episode?.events) + .map((event) => { + const eventId = nonEmptyString(event?.event_id); + const description = nonEmptyString(event?.description); + if (!eventId || !description) return null; + return { + eventId, + description, + device: nonEmptyString(event?.device), + modality: nonEmptyString(event?.modality) ?? 'event', + location: nonEmptyString(event?.location), + source: nonEmptyString(event?.source) ?? 'unknown', + timestamp: nonEmptyString(event?.timestamp), + characters: asArray(event?.characters).map(String), + }; + }) + .filter(Boolean); + if (!episodeId || events.length === 0) return null; + return { episodeId, events }; + }) + .filter(Boolean); + const questions = asArray(scenario?.questions) + .map((question) => { + const questionId = nonEmptyString(question?.question_id); + const text = nonEmptyString(question?.question); + const evidenceEventIds = [ + ...new Set(asArray(question?.evidence_event_ids).map(String).filter(Boolean)), + ]; + if (!questionId || !text || evidenceEventIds.length === 0) return null; + const checklist = asArray(question?.answer_checklist) + .map((entry) => ({ + point: nonEmptyString(entry?.point) ?? '', + sourceEvents: [ + ...new Set(asArray(entry?.source_events).map(String).filter(Boolean)), + ], + })) + .filter((entry) => entry.sourceEvents.length > 0); + return { + questionId, + scenarioId, + question: text, + evidenceEventIds, + checklist, + dimension: nonEmptyString(question?.dimension) ?? 'unknown', + questionUser: nonEmptyString(question?.question_user), + questionTime: nonEmptyString(question?.question_time), + questionDevice: nonEmptyString(question?.question_device), + }; + }) + .filter(Boolean); + if (episodes.length === 0 || questions.length === 0) return null; + return { + scenarioId, + description: nonEmptyString(scenario?.description), + timeSpan: nonEmptyString(scenario?.time_span), + episodes, + questions, + }; + }) + .filter(Boolean); + + if (scenarios.length === 0) { + return { ok: false, reason: 'dataset_has_no_usable_scenarios' }; + } + + const eventCount = scenarios.reduce( + (total, scenario) => + total + scenario.episodes.reduce((sum, episode) => sum + episode.events.length, 0), + 0, + ); + const questionCount = scenarios.reduce( + (total, scenario) => total + scenario.questions.length, + 0, + ); + + return { + ok: true, + dataset: { + scenarios, + stats: { scenarioCount: scenarios.length, eventCount, questionCount }, + }, + }; +} + +export async function loadMemFuseDataset({ + datasetPath = null, + env = process.env, + readFile = fsReadFile, +} = {}) { + const resolvedPath = datasetPath ? path.resolve(datasetPath) : resolveMemFuseDatasetPath(env); + let payload; + try { + payload = await readFile(resolvedPath, 'utf8'); + } catch (err) { + return { + available: false, + path: resolvedPath, + reason: err?.code === 'ENOENT' ? 'dataset_not_found' : 'dataset_read_failed', + error: err instanceof Error ? err.message : String(err), + }; + } + let raw; + try { + raw = JSON.parse(payload); + } catch (err) { + return { + available: false, + path: resolvedPath, + reason: 'dataset_parse_failed', + error: err instanceof Error ? err.message : String(err), + }; + } + const normalized = normalizeMemFuseDataset(raw); + if (!normalized.ok) { + return { available: false, path: resolvedPath, reason: normalized.reason }; + } + return { available: true, path: resolvedPath, ...normalized.dataset }; +} + +/** + * Flatten a scenario into rows shaped like the `memory_embeddings` table so the + * pgvector backend can consume them unchanged. + * + * `includeSourceTags` prepends `[device · location]`. It defaults to false so the + * baseline measures retrieval over raw event text rather than over a prefix + * format chosen here. + */ +export function buildScenarioCorpus(scenario, { includeSourceTags = false } = {}) { + const rows = []; + const byId = new Map(); + for (const episode of asArray(scenario?.episodes)) { + for (const event of asArray(episode?.events)) { + const tags = [event.device, event.location].filter(Boolean).join(' · '); + const content = + includeSourceTags && tags ? `[${tags}] ${event.description}` : event.description; + const timestampMs = event.timestamp ? Date.parse(event.timestamp) : NaN; + const isoTimestamp = Number.isFinite(timestampMs) + ? new Date(timestampMs).toISOString() + : null; + const row = { + id: event.eventId, + content, + type: event.modality, + created_at: isoTimestamp, + updated_at: isoTimestamp, + episodeId: episode.episodeId, + source: event.source, + device: event.device, + location: event.location, + timestampMs: Number.isFinite(timestampMs) ? timestampMs : 0, + }; + rows.push(row); + byId.set(row.id, row); + } + } + return { rows, byId }; +} + +export function buildScenarioCases( + scenario, + { dimensions = null, maxQuestions = null, questionIds = null } = {}, +) { + const dimensionFilter = dimensions?.length ? new Set(dimensions) : null; + const idFilter = questionIds?.length ? new Set(questionIds) : null; + const cases = asArray(scenario?.questions).filter((question) => { + if (dimensionFilter && !dimensionFilter.has(question.dimension)) return false; + if (idFilter && !idFilter.has(question.questionId)) return false; + return true; + }); + const capped = Number(maxQuestions); + return Number.isFinite(capped) && capped > 0 ? cases.slice(0, capped) : cases; +} + +const CJK_RANGE = /[\u4e00-\u9fff]/u; + +function tokenizeForEmbedding(text) { + const normalized = String(text ?? '') + .normalize('NFKC') + .toLowerCase(); + const tokens = []; + for (const match of normalized.matchAll(/[a-z0-9]{2,}/g)) tokens.push(match[0]); + if (CJK_RANGE.test(normalized)) { + const cjk = normalized.replace(/[^\u4e00-\u9fff]/gu, ''); + for (let index = 0; index + 2 <= cjk.length; index += 1) { + tokens.push(cjk.slice(index, index + 2)); + } + } + return tokens; +} + +function hashToken(token, dimensions) { + let hash = 2166136261; + for (let index = 0; index < token.length; index += 1) { + hash ^= token.charCodeAt(index); + hash = Math.imul(hash, 16777619); + } + return Math.abs(hash) % dimensions; +} + +/** + * Deterministic offline stand-in for a sentence embedder: hashed bag-of-tokens, + * L2 normalized. Cosine over these vectors approximates lexical overlap, NOT + * semantic similarity, so reports built on it are labelled `lexical-hash`. Pass + * a real `embedText` to measure semantic recall. + */ +export function createLexicalHashEmbedder({ dimensions = 256 } = {}) { + const size = Math.max(16, Math.min(4096, Number(dimensions) || 256)); + return function embedText(text) { + const vector = new Array(size).fill(0); + const tokens = tokenizeForEmbedding(text); + if (tokens.length === 0) return vector; + for (const token of tokens) { + vector[hashToken(token, size)] += 1; + } + let norm = 0; + for (const value of vector) norm += value * value; + norm = Math.sqrt(norm); + if (norm === 0) return vector; + for (let index = 0; index < size; index += 1) vector[index] /= norm; + return vector; + }; +} + +function cosineSimilarity(left, right) { + const length = Math.min(left.length, right.length); + let dot = 0; + let leftNorm = 0; + let rightNorm = 0; + for (let index = 0; index < length; index += 1) { + dot += left[index] * right[index]; + leftNorm += left[index] * left[index]; + rightNorm += right[index] * right[index]; + } + if (leftNorm === 0 || rightNorm === 0) return 0; + return dot / (Math.sqrt(leftNorm) * Math.sqrt(rightNorm)); +} + +function parseVectorLiteral(literal) { + const text = String(literal ?? '').trim(); + if (!text.startsWith('[') || !text.endsWith(']')) return null; + const inner = text.slice(1, -1); + if (!inner) return []; + const parts = inner.split(',').map((part) => Number(part)); + return parts.some((part) => !Number.isFinite(part)) ? null : parts; +} + +function likeParamToTerm(param) { + return String(param ?? '') + .replace(/^%+/, '') + .replace(/%+$/, '') + .toLowerCase(); +} + +/** + * Fake pg pool that emulates the two query shapes issued by + * `createPgvectorMemoryBackend.resolve`: the vector/recent candidate union and + * the `content ILIKE` keyword fallback. Emulating the SQL rather than bypassing + * it keeps the production ranking path under test. + */ +export function createCorpusPool({ rows, embedText, embeddingCache = new Map() }) { + const corpus = asArray(rows); + const byRecency = [...corpus].sort((left, right) => right.timestampMs - left.timestampMs); + + function embeddingFor(row) { + if (!embeddingCache.has(row.id)) { + embeddingCache.set(row.id, embedText(row.content)); + } + return embeddingCache.get(row.id); + } + + function toResultRow(row, score) { + return { + id: row.id, + content: row.content, + type: row.type, + created_at: row.created_at, + updated_at: row.updated_at, + score, + }; + } + + let vectorQueryCount = 0; + let keywordQueryCount = 0; + // Every id handed back to the backend, i.e. the candidate pool that ranking + // then narrows to top-k. Tracking it separates "candidate generation lost the + // evidence" from "ranking lost the evidence". + const returnedIds = new Set(); + + function trackRows(rows) { + for (const row of rows) returnedIds.add(row.id); + return rows; + } + + return { + stats: () => ({ + vectorQueryCount, + keywordQueryCount, + corpusSize: corpus.length, + candidateIds: new Set(returnedIds), + }), + async query(sql, params = []) { + const text = String(sql ?? ''); + if (text.includes('vector_candidates')) { + vectorQueryCount += 1; + const queryVector = parseVectorLiteral(params[1]); + const limit = Math.max(1, Number(params[2]) || 50); + const scored = corpus.map((row) => ({ + row, + score: queryVector ? cosineSimilarity(queryVector, embeddingFor(row)) : 0, + })); + const scoreById = new Map(scored.map((entry) => [entry.row.id, entry.score])); + const vectorTop = [...scored] + .sort((left, right) => right.score - left.score) + .slice(0, limit) + .map((entry) => entry.row); + const recentTop = byRecency.slice(0, limit); + const merged = new Map(); + for (const row of [...vectorTop, ...recentTop]) { + if (!merged.has(row.id)) merged.set(row.id, row); + } + return { + rows: trackRows( + [...merged.values()].map((row) => toResultRow(row, scoreById.get(row.id) ?? 0)), + ), + }; + } + if (text.includes('ILIKE')) { + keywordQueryCount += 1; + const limit = Math.max(1, Number(params[params.length - 1]) || 20); + const terms = params + .slice(1, params.length - 1) + .map(likeParamToTerm) + .filter(Boolean); + if (terms.length === 0) return { rows: [] }; + const matched = byRecency.filter((row) => { + const haystack = row.content.toLowerCase(); + return terms.some((term) => haystack.includes(term)); + }); + return { rows: trackRows(matched.slice(0, limit).map((row) => toResultRow(row, 1))) }; + } + return { rows: [] }; + }, + }; +} + +function checklistCoverage(checklist, retrievedIds) { + if (!checklist?.length) return null; + let covered = 0; + for (const entry of checklist) { + if (entry.sourceEvents.some((eventId) => retrievedIds.has(eventId))) covered += 1; + } + return covered / checklist.length; +} + +export async function runMemFuseBenchCase({ + testCase, + corpus, + embedText, + limit = 20, + candidateLimit = 100, + embeddingCache = new Map(), +}) { + const pool = createCorpusPool({ rows: corpus.rows, embedText, embeddingCache }); + const backend = createPgvectorMemoryBackend({ enabled: true, embedQuery: embedText, pool }); + const result = await backend.resolve({ + userId: 'memfuse-bench-user', + query: testCase.question, + limit, + candidateLimit, + }); + + const retrieved = result.memories.map((memory) => memory.id).filter(Boolean); + const retrievedIds = new Set(retrieved); + const gold = new Set(testCase.evidenceEventIds); + const hits = retrieved.filter((id) => gold.has(id)); + const firstHitIndex = retrieved.findIndex((id) => gold.has(id)); + const distractors = retrieved.filter((id) => + MEMFUSE_DISTRACTOR_SOURCES.includes(corpus.byId.get(id)?.source), + ); + + const { candidateIds } = pool.stats(); + const candidateHits = [...gold].filter((id) => candidateIds.has(id)); + const candidateRecall = gold.size > 0 ? candidateHits.length / gold.size : 0; + const recall = gold.size > 0 ? hits.length / gold.size : 0; + + return { + questionId: testCase.questionId, + scenarioId: testCase.scenarioId, + dimension: testCase.dimension, + goldCount: gold.size, + returned: retrieved.length, + hitCount: hits.length, + recall, + precision: retrieved.length > 0 ? hits.length / retrieved.length : 0, + hitAny: hits.length > 0, + candidateCount: candidateIds.size, + candidateHitCount: candidateHits.length, + candidateRecall, + // Positive value = evidence reached the candidate pool but ranking dropped + // it. This is the split that decides whether to fix retrieval or ranking. + rankingLoss: Math.max(0, candidateRecall - recall), + firstHitRank: firstHitIndex >= 0 ? firstHitIndex + 1 : null, + reciprocalRank: firstHitIndex >= 0 ? 1 / (firstHitIndex + 1) : 0, + checklistCoverage: checklistCoverage(testCase.checklist, retrievedIds), + distractorCount: distractors.length, + }; +} + +function mean(values) { + const usable = values.filter((value) => Number.isFinite(value)); + if (usable.length === 0) return 0; + return usable.reduce((total, value) => total + value, 0) / usable.length; +} + +function aggregate(results) { + const coverage = results + .map((item) => item.checklistCoverage) + .filter((value) => Number.isFinite(value)); + return { + caseCount: results.length, + recallAtK: mean(results.map((item) => item.recall)), + precisionAtK: mean(results.map((item) => item.precision)), + candidateRecall: mean(results.map((item) => item.candidateRecall)), + rankingLoss: mean(results.map((item) => item.rankingLoss)), + hitAnyRate: results.length + ? results.filter((item) => item.hitAny).length / results.length + : 0, + checklistCoverage: coverage.length ? mean(coverage) : null, + mrr: mean(results.map((item) => item.reciprocalRank)), + distractorRate: results.length + ? mean(results.map((item) => (item.returned ? item.distractorCount / item.returned : 0))) + : 0, + }; +} + +export async function runMemFuseBench({ + dataset, + scenarioIds = null, + dimensions = null, + maxQuestionsPerScenario = null, + questionIds = null, + limit = 20, + candidateLimit = 100, + embedText = null, + includeSourceTags = false, + onProgress = null, +} = {}) { + const scenarios = asArray(dataset?.scenarios); + if (scenarios.length === 0) throw new Error('runMemFuseBench requires a loaded dataset'); + const embedder = typeof embedText === 'function' ? embedText : createLexicalHashEmbedder(); + const embeddingMode = typeof embedText === 'function' ? 'external' : 'lexical-hash'; + const scenarioFilter = scenarioIds?.length ? new Set(scenarioIds) : null; + // The pgvector backend clamps its own limit to 50; mirror it so the reported + // `limit` never overstates how many rows were actually considered. + const effectiveLimit = Math.max(1, Math.min(50, Number(limit) || 20)); + + const results = []; + const scenarioReports = []; + for (const scenario of scenarios) { + if (scenarioFilter && !scenarioFilter.has(scenario.scenarioId)) continue; + const cases = buildScenarioCases(scenario, { + dimensions, + maxQuestions: maxQuestionsPerScenario, + questionIds, + }); + if (cases.length === 0) continue; + const corpus = buildScenarioCorpus(scenario, { includeSourceTags }); + // One cache per scenario: corpus embeddings are reused across that + // scenario's questions, which is where nearly all the cost sits. + const embeddingCache = new Map(); + const scenarioResults = []; + for (const testCase of cases) { + const caseResult = await runMemFuseBenchCase({ + testCase, + corpus, + embedText: embedder, + limit: effectiveLimit, + candidateLimit, + embeddingCache, + }); + scenarioResults.push(caseResult); + results.push(caseResult); + if (typeof onProgress === 'function') { + onProgress({ scenarioId: scenario.scenarioId, completed: results.length, ...caseResult }); + } + } + scenarioReports.push({ + scenarioId: scenario.scenarioId, + corpusSize: corpus.rows.length, + ...aggregate(scenarioResults), + }); + } + + if (results.length === 0) throw new Error('runMemFuseBench selected zero questions'); + + const byDimension = {}; + for (const dimension of new Set(results.map((item) => item.dimension))) { + byDimension[dimension] = aggregate(results.filter((item) => item.dimension === dimension)); + } + + return { + limit: effectiveLimit, + candidateLimit, + embeddingMode, + includeSourceTags, + overall: aggregate(results), + byScenario: scenarioReports, + byDimension, + results, + }; +} + +export function summarizeMemFuseBench(report) { + const round = (value) => (Number.isFinite(value) ? Number(value.toFixed(4)) : null); + return { + limit: report.limit, + embeddingMode: report.embeddingMode, + caseCount: report.overall.caseCount, + recallAtK: round(report.overall.recallAtK), + precisionAtK: round(report.overall.precisionAtK), + candidateRecall: round(report.overall.candidateRecall), + rankingLoss: round(report.overall.rankingLoss), + hitAnyRate: round(report.overall.hitAnyRate), + checklistCoverage: round(report.overall.checklistCoverage), + mrr: round(report.overall.mrr), + distractorRate: round(report.overall.distractorRate), + weakestDimension: + Object.entries(report.byDimension) + .sort((left, right) => left[1].recallAtK - right[1].recallAtK) + .map(([dimension]) => dimension)[0] ?? null, + }; +} diff --git a/memory-v2-memfuse-bench.test.mjs b/memory-v2-memfuse-bench.test.mjs new file mode 100644 index 0000000..cb44769 --- /dev/null +++ b/memory-v2-memfuse-bench.test.mjs @@ -0,0 +1,449 @@ +import assert from 'node:assert/strict'; +import test from 'node:test'; + +import { + MEMFUSE_DATASET_ENV, + buildScenarioCases, + buildScenarioCorpus, + createCorpusPool, + createLexicalHashEmbedder, + loadMemFuseDataset, + normalizeMemFuseDataset, + resolveMemFuseDatasetPath, + runMemFuseBench, + runMemFuseBenchCase, + summarizeMemFuseBench, +} from './memory-v2-memfuse-bench.mjs'; + +// Mirrors the real MemFuseBench shape (scenarios → episodes → events, plus +// questions carrying evidence_event_ids and answer_checklist) at a size that +// keeps this suite hermetic: the 6MB upstream dataset is never required. +function makeRawDataset() { + return { + metadata: { total_questions: 3 }, + scenarios: [ + { + scenario_id: 'sc1', + description: 'Scenario sc1', + time_span: '2026-05-11 ~ 2026-05-12', + episodes: [ + { + episode_id: 'sc1_ep1', + events: [ + { + event_id: 'sc1_ep1_e1', + device: 'thermostat_living', + modality: 'environment', + characters: [], + description: 'Living room thermostat reading: temperature 21.8C, humidity 47%.', + timestamp: '2026-05-11T06:00:00', + location: 'living_room', + source: 'periodic', + }, + { + event_id: 'sc1_ep1_e2', + device: 'purifier_living', + modality: 'appliance', + characters: ['Sarah'], + description: 'Sarah switched the air purifier to turbo because of pollen.', + timestamp: '2026-05-11T08:12:00', + location: 'living_room', + source: 'storyline', + }, + { + event_id: 'sc1_ep1_e3', + device: 'curtain_living', + modality: 'appliance', + characters: ['Sarah'], + description: 'Smart curtains closed to reduce pollen entering the living room.', + timestamp: '2026-05-11T08:15:00', + location: 'living_room', + source: 'storyline', + }, + { + event_id: 'sc1_ep1_e4', + device: 'phone_david', + modality: 'message', + characters: ['David'], + description: 'Unrelated chatter about a weekend hiking trip to the coast.', + timestamp: '2026-05-11T09:00:00', + location: 'office', + source: 'noise', + }, + { + event_id: 'sc1_ep1_e5', + device: 'speaker_kitchen', + modality: 'audio', + characters: ['Ethan'], + description: 'Adversarial lookalike: a podcast episode discussing pollen forecasts.', + timestamp: '2026-05-11T09:30:00', + location: 'kitchen', + source: 'adversarial', + }, + ], + }, + ], + questions: [ + { + question_id: 'q1', + question: 'Why did Sarah change the purifier and curtains for pollen?', + answer: 'Because of pollen.', + answer_checklist: [ + { point: 'purifier to turbo', source_events: ['sc1_ep1_e2'] }, + { point: 'curtains closed', source_events: ['sc1_ep1_e3'] }, + ], + evidence_event_ids: ['sc1_ep1_e2', 'sc1_ep1_e3'], + dimension: 'cross_device_causal_reasoning', + question_user: 'David', + question_time: '2026-05-11T12:30:00', + question_device: 'phone_david', + }, + { + question_id: 'q2', + question: 'What was the living room thermostat temperature reading?', + answer: '21.8C', + answer_checklist: [{ point: 'temperature', source_events: ['sc1_ep1_e1'] }], + evidence_event_ids: ['sc1_ep1_e1'], + dimension: 'cross_device_information_fusion', + question_user: 'Sarah', + question_time: '2026-05-11T13:00:00', + question_device: 'tablet_home', + }, + ], + }, + { + scenario_id: 'sc2', + episodes: [ + { + episode_id: 'sc2_ep1', + events: [ + { + event_id: 'sc2_ep1_e1', + device: 'watch_ethan', + modality: 'wearable', + characters: ['Ethan'], + description: 'Wearable recorded an elevated heart rate during the afternoon run.', + timestamp: '2026-05-12T17:05:00', + location: 'park', + source: 'storyline', + }, + ], + }, + ], + questions: [ + { + question_id: 'q1', + question: 'What did the wearable record during the afternoon run?', + answer: 'Elevated heart rate.', + answer_checklist: [{ point: 'elevated heart rate', source_events: ['sc2_ep1_e1'] }], + evidence_event_ids: ['sc2_ep1_e1'], + dimension: 'perspective_difference', + question_user: 'Sarah', + question_time: '2026-05-12T19:00:00', + question_device: 'phone_sarah', + }, + ], + }, + ], + }; +} + +function loadFixtureDataset() { + const normalized = normalizeMemFuseDataset(makeRawDataset()); + assert.equal(normalized.ok, true); + return normalized.dataset; +} + +test('resolveMemFuseDatasetPath prefers the explicit env override', () => { + const explicit = resolveMemFuseDatasetPath({ [MEMFUSE_DATASET_ENV]: '/tmp/custom/bench.json' }); + assert.equal(explicit, '/tmp/custom/bench.json'); + + const viaRoot = resolveMemFuseDatasetPath({ MEMFUSE_PROJECT_ROOT: '/srv/refs' }); + assert.equal(viaRoot, '/srv/refs/mi-memory/MemFuse/MemFuseBench/memfusebench_dataset.json'); +}); + +test('normalizeMemFuseDataset keeps usable scenarios and reports stats', () => { + const normalized = normalizeMemFuseDataset(makeRawDataset()); + assert.equal(normalized.ok, true); + assert.deepEqual(normalized.dataset.stats, { + scenarioCount: 2, + eventCount: 6, + questionCount: 3, + }); + const [scenario] = normalized.dataset.scenarios; + assert.equal(scenario.scenarioId, 'sc1'); + assert.equal(scenario.questions[0].evidenceEventIds.length, 2); + assert.equal(scenario.questions[0].checklist.length, 2); +}); + +test('normalizeMemFuseDataset drops events and questions that cannot be scored', () => { + const raw = makeRawDataset(); + raw.scenarios[0].episodes[0].events.push({ event_id: 'no_description' }); + raw.scenarios[0].questions.push({ + question_id: 'q_no_evidence', + question: 'Unscoreable question', + evidence_event_ids: [], + }); + const normalized = normalizeMemFuseDataset(raw); + assert.equal(normalized.ok, true); + assert.equal(normalized.dataset.stats.eventCount, 6); + assert.equal(normalized.dataset.stats.questionCount, 3); +}); + +test('normalizeMemFuseDataset rejects payloads with no usable scenario', () => { + assert.equal(normalizeMemFuseDataset({}).reason, 'dataset_has_no_usable_scenarios'); + assert.equal(normalizeMemFuseDataset({ scenarios: [{}] }).reason, 'dataset_has_no_usable_scenarios'); +}); + +test('loadMemFuseDataset degrades instead of throwing when the dataset is absent', async () => { + const missing = await loadMemFuseDataset({ + datasetPath: '/tmp/definitely-absent.json', + readFile: async () => { + const err = new Error('ENOENT'); + err.code = 'ENOENT'; + throw err; + }, + }); + assert.equal(missing.available, false); + assert.equal(missing.reason, 'dataset_not_found'); + + const unparsable = await loadMemFuseDataset({ + datasetPath: '/tmp/broken.json', + readFile: async () => 'not json', + }); + assert.equal(unparsable.available, false); + assert.equal(unparsable.reason, 'dataset_parse_failed'); +}); + +test('loadMemFuseDataset returns normalized scenarios on success', async () => { + const loaded = await loadMemFuseDataset({ + datasetPath: '/tmp/fixture.json', + readFile: async () => JSON.stringify(makeRawDataset()), + }); + assert.equal(loaded.available, true); + assert.equal(loaded.path, '/tmp/fixture.json'); + assert.equal(loaded.stats.questionCount, 3); +}); + +test('buildScenarioCorpus emits pgvector-shaped rows and can prepend source tags', () => { + const [scenario] = loadFixtureDataset().scenarios; + const plain = buildScenarioCorpus(scenario); + assert.equal(plain.rows.length, 5); + const first = plain.byId.get('sc1_ep1_e1'); + assert.equal(first.content, 'Living room thermostat reading: temperature 21.8C, humidity 47%.'); + assert.equal(first.type, 'environment'); + assert.equal(first.source, 'periodic'); + assert.equal(first.episodeId, 'sc1_ep1'); + assert.equal(first.created_at, new Date('2026-05-11T06:00:00').toISOString()); + + const tagged = buildScenarioCorpus(scenario, { includeSourceTags: true }); + assert.equal( + tagged.byId.get('sc1_ep1_e1').content, + '[thermostat_living · living_room] Living room thermostat reading: temperature 21.8C, humidity 47%.', + ); +}); + +test('buildScenarioCases filters by dimension, id and cap', () => { + const [scenario] = loadFixtureDataset().scenarios; + assert.equal(buildScenarioCases(scenario).length, 2); + assert.deepEqual( + buildScenarioCases(scenario, { dimensions: ['cross_device_causal_reasoning'] }).map( + (item) => item.questionId, + ), + ['q1'], + ); + assert.deepEqual( + buildScenarioCases(scenario, { questionIds: ['q2'] }).map((item) => item.questionId), + ['q2'], + ); + assert.equal(buildScenarioCases(scenario, { maxQuestions: 1 }).length, 1); +}); + +test('createLexicalHashEmbedder is deterministic and L2 normalized', () => { + const embed = createLexicalHashEmbedder({ dimensions: 64 }); + const left = embed('pollen purifier curtains'); + const right = embed('pollen purifier curtains'); + assert.deepEqual(left, right); + assert.equal(left.length, 64); + const norm = Math.sqrt(left.reduce((total, value) => total + value * value, 0)); + assert.ok(Math.abs(norm - 1) < 1e-9); + assert.notDeepEqual(embed('completely different text'), left); + assert.deepEqual( + embed(''), + new Array(64).fill(0), + ); +}); + +test('createCorpusPool emulates the vector union and the ILIKE fallback', async () => { + const [scenario] = loadFixtureDataset().scenarios; + const corpus = buildScenarioCorpus(scenario); + const embed = createLexicalHashEmbedder({ dimensions: 128 }); + const pool = createCorpusPool({ rows: corpus.rows, embedText: embed }); + + const vectorSql = 'WITH vector_candidates AS (SELECT ...) SELECT DISTINCT ON (id) ...'; + const vectorResult = await pool.query(vectorSql, [ + 'user', + `[${embed('pollen purifier').join(',')}]`, + 3, + ]); + assert.ok(vectorResult.rows.length > 0); + assert.ok(vectorResult.rows.every((row) => typeof row.score === 'number')); + assert.ok(vectorResult.rows.some((row) => row.id === 'sc1_ep1_e2')); + + const keywordResult = await pool.query( + 'SELECT id, content FROM memory_embeddings WHERE user_id = $1 AND (content ILIKE $2) LIMIT $3', + ['user', '%curtains%', 10], + ); + assert.deepEqual( + keywordResult.rows.map((row) => row.id), + ['sc1_ep1_e3'], + ); + + const unknown = await pool.query('SELECT 1', []); + assert.deepEqual(unknown.rows, []); + assert.equal(pool.stats().corpusSize, 5); + assert.equal(pool.stats().vectorQueryCount, 1); + assert.equal(pool.stats().keywordQueryCount, 1); +}); + +test('runMemFuseBenchCase scores recall, checklist coverage and distractors', async () => { + const [scenario] = loadFixtureDataset().scenarios; + const corpus = buildScenarioCorpus(scenario); + const [testCase] = scenario.questions; + const result = await runMemFuseBenchCase({ + testCase, + corpus, + embedText: createLexicalHashEmbedder({ dimensions: 256 }), + limit: 5, + }); + + assert.equal(result.questionId, 'q1'); + assert.equal(result.dimension, 'cross_device_causal_reasoning'); + assert.equal(result.goldCount, 2); + assert.equal(result.hitCount, 2); + assert.equal(result.recall, 1); + assert.equal(result.hitAny, true); + assert.equal(result.checklistCoverage, 1); + assert.ok(result.firstHitRank >= 1); + assert.ok(result.reciprocalRank > 0); + assert.ok(result.precision > 0 && result.precision <= 1); + // e4 (noise) and e5 (adversarial) are the only planted distractors. + assert.ok(result.distractorCount <= 2); + // Small corpus: every row reaches the candidate pool, so ranking loses nothing. + assert.equal(result.candidateRecall, 1); + assert.equal(result.rankingLoss, 0); + assert.equal(result.candidateCount, 5); +}); + +test('runMemFuseBenchCase separates candidate recall from ranking loss', async () => { + const [scenario] = loadFixtureDataset().scenarios; + const corpus = buildScenarioCorpus(scenario); + const [testCase] = scenario.questions; + // limit 1 forces ranking to drop evidence that candidate generation found. + const result = await runMemFuseBenchCase({ + testCase, + corpus, + embedText: createLexicalHashEmbedder({ dimensions: 256 }), + limit: 1, + }); + assert.equal(result.candidateRecall, 1); + assert.ok(result.recall < 1); + assert.ok(result.rankingLoss > 0); + assert.equal(result.rankingLoss, result.candidateRecall - result.recall); +}); + +test('runMemFuseBenchCase reports a clean miss when nothing relevant exists', async () => { + const [scenario] = loadFixtureDataset().scenarios; + const corpus = buildScenarioCorpus(scenario); + const result = await runMemFuseBenchCase({ + testCase: { + questionId: 'synthetic-miss', + scenarioId: 'sc1', + dimension: 'cross_user_query', + question: 'zzz nonexistent topic zzz', + evidenceEventIds: ['sc1_ep1_missing'], + checklist: [{ point: 'absent', sourceEvents: ['sc1_ep1_missing'] }], + }, + corpus, + embedText: createLexicalHashEmbedder({ dimensions: 256 }), + limit: 5, + }); + assert.equal(result.hitCount, 0); + assert.equal(result.recall, 0); + assert.equal(result.hitAny, false); + assert.equal(result.firstHitRank, null); + assert.equal(result.reciprocalRank, 0); + assert.equal(result.checklistCoverage, 0); +}); + +test('runMemFuseBench aggregates overall, per-scenario and per-dimension metrics', async () => { + const dataset = loadFixtureDataset(); + const report = await runMemFuseBench({ dataset, limit: 5 }); + + assert.equal(report.embeddingMode, 'lexical-hash'); + assert.equal(report.limit, 5); + assert.equal(report.overall.caseCount, 3); + assert.ok(report.overall.recallAtK > 0); + assert.ok(report.overall.hitAnyRate > 0); + assert.deepEqual( + report.byScenario.map((item) => item.scenarioId), + ['sc1', 'sc2'], + ); + assert.equal(report.byScenario[0].corpusSize, 5); + assert.ok(Object.keys(report.byDimension).length >= 2); + assert.equal(report.results.length, 3); +}); + +test('runMemFuseBench honours scenario, dimension and cap filters', async () => { + const dataset = loadFixtureDataset(); + const report = await runMemFuseBench({ + dataset, + scenarioIds: ['sc1'], + dimensions: ['cross_device_information_fusion'], + limit: 5, + }); + assert.equal(report.overall.caseCount, 1); + assert.equal(report.results[0].questionId, 'q2'); + + const capped = await runMemFuseBench({ dataset, maxQuestionsPerScenario: 1, limit: 5 }); + assert.equal(capped.overall.caseCount, 2); +}); + +test('runMemFuseBench clamps limit to the pgvector backend ceiling', async () => { + const report = await runMemFuseBench({ dataset: loadFixtureDataset(), limit: 500 }); + assert.equal(report.limit, 50); +}); + +test('runMemFuseBench marks an injected embedder as external and reports progress', async () => { + const dataset = loadFixtureDataset(); + const seen = []; + const report = await runMemFuseBench({ + dataset, + limit: 5, + embedText: createLexicalHashEmbedder({ dimensions: 32 }), + onProgress: (event) => seen.push(event.questionId), + }); + assert.equal(report.embeddingMode, 'external'); + assert.equal(seen.length, 3); +}); + +test('runMemFuseBench rejects an empty dataset or an empty selection', async () => { + await assert.rejects(() => runMemFuseBench({ dataset: { scenarios: [] } }), /requires a loaded dataset/); + await assert.rejects( + () => runMemFuseBench({ dataset: loadFixtureDataset(), dimensions: ['not_a_dimension'] }), + /selected zero questions/, + ); +}); + +test('summarizeMemFuseBench rounds metrics and names the weakest dimension', async () => { + const report = await runMemFuseBench({ dataset: loadFixtureDataset(), limit: 5 }); + const summary = summarizeMemFuseBench(report); + assert.equal(summary.caseCount, 3); + assert.equal(summary.embeddingMode, 'lexical-hash'); + assert.ok(summary.recallAtK >= 0 && summary.recallAtK <= 1); + assert.ok(summary.candidateRecall >= summary.recallAtK); + assert.ok(summary.rankingLoss >= 0); + assert.ok(summary.mrr >= 0 && summary.mrr <= 1); + assert.ok(summary.distractorRate >= 0 && summary.distractorRate <= 1); + assert.ok(typeof summary.weakestDimension === 'string'); +}); diff --git a/package.json b/package.json index 043da3a..1e842f3 100644 --- a/package.json +++ b/package.json @@ -69,6 +69,8 @@ "repair:memory-v2-candidates": "node scripts/repair-memory-v2-candidates.mjs", "verify:memory-v2-shadow-audit": "node --test memory-v2-shadow-audit.test.mjs", "verify:memory-v2-recall-benchmark": "node --test memory-v2-recall-benchmark.test.mjs", + "verify:memory-v2-memfuse-bench": "node --test memory-v2-memfuse-bench.test.mjs", + "bench:memory-v2-memfuse": "node scripts/run-memory-v2-memfuse-bench.mjs", "trace:mindspace-artifact": "node scripts/trace-mindspace-artifact.mjs", "check:conversation-package-manifest": "node scripts/check-conversation-package-manifest.mjs", "check:memory-v2-phase-a": "node scripts/check-memory-v2-phase-a-ready.mjs", diff --git a/scripts/run-memory-v2-memfuse-bench.mjs b/scripts/run-memory-v2-memfuse-bench.mjs new file mode 100644 index 0000000..665d6e1 --- /dev/null +++ b/scripts/run-memory-v2-memfuse-bench.mjs @@ -0,0 +1,242 @@ +#!/usr/bin/env node +/** + * Run the MemFuseBench retrieval benchmark against the Memory V2 pgvector + * ranking path. + * + * The dataset lives outside this repository (MIT, Mi-Memory / Darwin Agent + * Team). Point at it with --dataset or MEMFUSE_BENCH_DATASET; without it the + * run reports `skipped` and exits 0 unless --strict is passed. + * + * Usage: + * node scripts/run-memory-v2-memfuse-bench.mjs + * node scripts/run-memory-v2-memfuse-bench.mjs --scenario sc1 --limit 20 + * node scripts/run-memory-v2-memfuse-bench.mjs --dimension multi_source_conflict_arbitration + * node scripts/run-memory-v2-memfuse-bench.mjs --embedding-module ./scripts/my-embedder.mjs + * node scripts/run-memory-v2-memfuse-bench.mjs --json --out .release-gate/memfuse.json + */ + +import fs from 'node:fs/promises'; +import path from 'node:path'; + +import { + MEMFUSE_DIMENSIONS, + loadMemFuseDataset, + runMemFuseBench, + summarizeMemFuseBench, +} from '../memory-v2-memfuse-bench.mjs'; +import { resolveEmbeddingModuleSpecifier } from '../memory-v2-recall-benchmark.mjs'; + +function parseArgs(argv) { + const options = { + dataset: null, + scenarios: null, + dimensions: null, + questionIds: null, + limit: 20, + candidateLimit: 100, + maxQuestions: null, + sourceTags: false, + embeddingModule: null, + json: false, + out: null, + strict: false, + quiet: false, + }; + const list = (value) => + String(value ?? '') + .split(',') + .map((item) => item.trim()) + .filter(Boolean); + + for (let index = 0; index < argv.length; index += 1) { + const arg = argv[index]; + const next = () => argv[(index += 1)]; + switch (arg) { + case '--dataset': options.dataset = next(); break; + case '--scenario': options.scenarios = list(next()); break; + case '--dimension': options.dimensions = list(next()); break; + case '--question': options.questionIds = list(next()); break; + case '--limit': options.limit = Number(next()); break; + case '--candidate-limit': options.candidateLimit = Number(next()); break; + case '--max-questions': options.maxQuestions = Number(next()); break; + case '--source-tags': options.sourceTags = true; break; + case '--embedding-module': options.embeddingModule = next(); break; + case '--json': options.json = true; break; + case '--out': options.out = next(); break; + case '--strict': options.strict = true; break; + case '--quiet': options.quiet = true; break; + case '-h': + case '--help': options.help = true; break; + default: + if (arg.startsWith('--')) throw new Error(`Unknown flag: ${arg}`); + } + } + return options; +} + +function usage() { + process.stdout.write(`Usage: node scripts/run-memory-v2-memfuse-bench.mjs [options] + + --dataset MemFuseBench dataset JSON (default: $MEMFUSE_BENCH_DATASET) + --scenario Comma-separated scenario ids, e.g. sc1,sc2 + --dimension Comma-separated dimensions. Known values: + ${MEMFUSE_DIMENSIONS.join('\n ')} + --question Comma-separated question ids + --limit Retrieved rows per question (clamped to 50) + --candidate-limit Candidate pool size before ranking (clamped to 100) + --max-questions Cap questions per scenario (useful for smoke runs) + --source-tags Prepend "[device · location]" to event text + --embedding-module Module exporting embedQuery/embedText/default + --json Print the full report as JSON + --out Write the JSON report to a file + --strict Exit non-zero when the dataset is unavailable + --quiet Suppress per-scenario progress +`); +} + +async function loadEmbedder(specifier) { + const resolved = resolveEmbeddingModuleSpecifier(specifier); + if (!resolved) throw new Error(`Cannot resolve embedding module: ${specifier}`); + const imported = await import(resolved); + const embed = imported?.embedText ?? imported?.embedQuery ?? imported?.default; + if (typeof embed !== 'function') { + throw new Error(`Embedding module must export embedText, embedQuery or default: ${specifier}`); + } + return embed; +} + +function formatPercent(value) { + return Number.isFinite(value) ? `${(value * 100).toFixed(1)}%` : 'n/a'; +} + +function formatRow(label, metrics, width) { + return [ + label.padEnd(width), + String(metrics.caseCount).padStart(5), + formatPercent(metrics.candidateRecall).padStart(9), + formatPercent(metrics.recallAtK).padStart(8), + formatPercent(metrics.rankingLoss).padStart(8), + formatPercent(metrics.hitAnyRate).padStart(8), + formatPercent(metrics.checklistCoverage).padStart(10), + (Number.isFinite(metrics.mrr) ? metrics.mrr.toFixed(3) : 'n/a').padStart(7), + formatPercent(metrics.distractorRate).padStart(11), + ].join(' '); +} + +function printReport(report) { + const labels = [ + ...report.byScenario.map((item) => item.scenarioId), + ...Object.keys(report.byDimension), + 'OVERALL', + ]; + const width = Math.max(12, ...labels.map((label) => label.length)); + const header = [ + 'group'.padEnd(width), + 'cases'.padStart(5), + 'cand.rec'.padStart(9), + 'recall'.padStart(8), + 'rankLoss'.padStart(8), + 'hit@k'.padStart(8), + 'checklist'.padStart(10), + 'mrr'.padStart(7), + 'distractor'.padStart(11), + ].join(' '); + + process.stdout.write(`\nMemFuseBench retrieval @k=${report.limit}`); + process.stdout.write( + ` · embedding=${report.embeddingMode}` + + ` · sourceTags=${report.includeSourceTags ? 'on' : 'off'}\n`, + ); + if (report.embeddingMode === 'lexical-hash') { + process.stdout.write( + 'NOTE: lexical-hash is a deterministic offline stand-in, not a semantic embedder.\n' + + ' Use --embedding-module to measure real semantic recall.\n', + ); + } + process.stdout.write(`\n${header}\n${'-'.repeat(header.length)}\n`); + for (const scenario of report.byScenario) { + process.stdout.write(`${formatRow(scenario.scenarioId, scenario, width)}\n`); + } + process.stdout.write(`${'-'.repeat(header.length)}\n`); + for (const [dimension, metrics] of Object.entries(report.byDimension).sort( + (left, right) => left[1].recallAtK - right[1].recallAtK, + )) { + process.stdout.write(`${formatRow(dimension, metrics, width)}\n`); + } + process.stdout.write(`${'-'.repeat(header.length)}\n`); + process.stdout.write(`${formatRow('OVERALL', report.overall, width)}\n\n`); +} + +async function main() { + const options = parseArgs(process.argv.slice(2)); + if (options.help) { + usage(); + return 0; + } + + const loaded = await loadMemFuseDataset({ datasetPath: options.dataset }); + if (!loaded.available) { + const message = + `MemFuseBench dataset unavailable (${loaded.reason}) at ${loaded.path}\n` + + 'Clone it outside this repo, then set MEMFUSE_BENCH_DATASET or pass --dataset:\n' + + ' git clone https://github.com/Darwin-Agent/Mi-Memory ~/Project/mi-memory\n'; + process.stderr.write(message); + return options.strict ? 1 : 0; + } + + if (!options.quiet) { + process.stdout.write( + `dataset: ${loaded.path}\n` + + `scenarios=${loaded.stats.scenarioCount} ` + + `events=${loaded.stats.eventCount} questions=${loaded.stats.questionCount}\n`, + ); + } + + const embedText = options.embeddingModule + ? await loadEmbedder(options.embeddingModule) + : null; + + let lastScenario = null; + const report = await runMemFuseBench({ + dataset: loaded, + scenarioIds: options.scenarios, + dimensions: options.dimensions, + questionIds: options.questionIds, + maxQuestionsPerScenario: Number.isFinite(options.maxQuestions) ? options.maxQuestions : null, + limit: options.limit, + candidateLimit: options.candidateLimit, + includeSourceTags: options.sourceTags, + embedText, + onProgress: options.quiet + ? null + : (event) => { + if (event.scenarioId !== lastScenario) { + lastScenario = event.scenarioId; + process.stderr.write(` running ${event.scenarioId}...\n`); + } + }, + }); + + if (options.json) { + process.stdout.write(`${JSON.stringify(report, null, 2)}\n`); + } else { + printReport(report); + process.stdout.write(`${JSON.stringify(summarizeMemFuseBench(report), null, 2)}\n`); + } + + if (options.out) { + const target = path.resolve(options.out); + await fs.mkdir(path.dirname(target), { recursive: true }); + await fs.writeFile(target, `${JSON.stringify(report, null, 2)}\n`, 'utf8'); + process.stderr.write(`report written to ${target}\n`); + } + + return 0; +} + +main() + .then((code) => process.exit(code)) + .catch((err) => { + process.stderr.write(`${err instanceof Error ? err.stack : String(err)}\n`); + process.exit(1); + });