Add MemFuseBench retrieval harness and first baseline report.

Wire an offline benchmark over the production pgvector ranking path so Memory V2 recall can be measured with candidate/ranking loss split before lifecycle work lands.

Co-authored-by: Cursor <cursoragent@cursor.com>

docs/architecture/memfuse-bench-baseline.md

@@ -0,0 +1,132 @@
+# MemFuseBench 检索基线（Memory V2）
+
+日期: 2026-09-02
+
+状态: 已落地测量工具 + 首份基线。**不改动任何生产运行时路径**，纯离线评测。
+
+## 0. 结论
+
+在 MemFuseBench 的 357 个证据锚定问题、7823 条来源标注事件上跑现网 pgvector 检索排序链路，得到：
+
+| 指标 | k=5 | k=20 | k=50 |
+|---|---|---|---|
+| 候选召回 candidateRecall | 36.6% | 36.6% | 36.6% |
+| 最终召回 recall@k | 6.8% | 13.5% | 22.5% |
+| **排序丢失 rankingLoss** | **29.8%** | **23.1%** | **14.2%** |
+| 至少命中一条 hitAny@k | 21.8% | 41.7% | 64.4% |
+| 清单覆盖 checklistCoverage | 7.8% | 16.8% | 28.3% |
+| MRR | 0.135 | 0.153 | 0.160 |
+| 干扰项占比 distractorRate | 8.5% | 10.8% | 12.6% |
+
+两个可操作结论：
+
+1. **天花板在候选生成，不在 k。** candidateRecall 恒为 36.6%，与 k 无关——超过 63% 的证据从未进入候选池，再怎么调排序或放大 k 都拿不回来。
+2. **进了候选池的证据，排序又丢掉了一大半。** k=20 时候选池已包含 36.6% 的证据，最终只留下 13.5%，rankingLoss 23.1 个百分点。即使 k=50（在约 200 条候选里返回 50 条），仍丢 14.2 个百分点。MRR 从 k=5 到 k=50 只从 0.135 涨到 0.160，说明列表头部的质量与 k 无关地差。
+
+按维度看，`multi_source_conflict_arbitration` 明显最好（候选 71.5% / 召回 39.4%），`cross_device_information_fusion` 最差（26.5% / 6.1%）。
+
+## 1. 两个已定位的机制
+
+### 机制一：`lexicalQueryCoverage` 是主排序键，但判别力接近随机
+
+`memory-v2-pgvector.mjs` 的 `rankHybridCandidates` 把 `lexicalScore` 作为**第一排序键**，向量分只在 lexical 分相同时才起作用。而 `lexicalQueryCoverage` 是字符 2-gram 覆盖率。全量 357 题实测：
+
+- gold 事件平均 coverage：`0.5366`
+- 非 gold 平均 coverage：`0.4853`
+- gold 高于语料中位数的比例：`64.5%`（随机为 50%）
+
+也就是一个只比随机好 14.5 个百分点的弱信号，被用作压倒性的主排序键。在英文语料上排名最高的往往是最长、bigram 最密的无关句子。
+
+这个启发式对中文是合理的——CJK 2-gram 近似于词，判别力强；对英文退化严重，因为 `th`/`he`/`in` 这类 bigram 在任何长句里都存在。
+
+### 机制二：keyword 回退路径按时间排序截断，不按相关性
+
+`fetchKeywordCandidates` 的 SQL 是 `... WHERE content ILIKE ANY(...) ORDER BY updated_at DESC LIMIT n`。全量实测：
+
+- ILIKE 能匹配到的 gold：`60.4%`
+- 按 `updated_at DESC` 截断 100 条后存活的 gold：`14.2%`
+- 至少存活一条 gold 的问题占比：`32.5%`
+
+一次纯粹由"按时间截断"造成的 46 个百分点损失。语料里匹配关键词的事件一旦多于 limit，这条路召回的就是"最新的匹配"而不是"最相关的匹配"。
+
+## 2. 必须一起读的三条限制
+
+不要把上面的绝对数字当成生产召回率。
+
+1. **默认嵌入是 `lexical-hash`，不是语义嵌入。** 它是哈希词袋 + L2 归一化的确定性离线替身，cosine 近似词面重叠。报告里 `embeddingMode` 会标出来。机制一和机制二与嵌入无关（都在 lexical / keyword 路径上），但 36.6% 这个候选天花板里有一部分要归因于替身嵌入弱。用 `--embedding-module` 接真实嵌入后才能给向量路径定论。
+2. **语料是英文，而 lexical 层是按中文调的。** 见机制一。这既是限制也是发现：一旦 Memind 要处理非中文内容，这条排序键就失效。
+3. **语料粒度比现网粗。** MemFuseBench 的语料是事件级（单场景 936–1886 条原始事件），而现网 `memory_embeddings` 存的是已经筛过的用户记忆，条数少得多。所以这份基线衡量的是"如果把原始事件流直接灌进现有检索层会怎样"——正好是 Event Kernel 方向会产生的形态。
+
+## 3. 数据集与授权
+
+**数据集不入库。** 它在仓库外，通过 `MEMFUSE_BENCH_DATASET` 或 `--dataset` 指定；缺失时所有入口返回 `{ available: false, reason }`，CLI 打印 skip 并 exit 0（`--strict` 才 exit 1）。CI 无需数据集即可全绿。
+
+```bash
+git clone https://github.com/Darwin-Agent/Mi-Memory ~/Project/mi-memory
+```
+
+默认路径 `~/Project/mi-memory/MemFuse/MemFuseBench/memfusebench_dataset.json`（6.2MB）。
+
+| 项目 | 授权 | 本仓库可以做什么 |
+|---|---|---|
+| Mi-Memory / MemFuseBench | MIT（Darwin Agent Team, Xiaomi） | 可自由使用数据集与论文思想 |
+| JKRiver / Riverse | **AGPL-3.0 或商业双授权** | **只做架构级参考，禁止移植代码** |
+
+JKRiver 的 `LICENSE` 明确把"集成进专有产品"和"作为商业 SaaS 提供而不开源修改"列为需要单独商业授权的场景。Memind 是带计费的商业 SaaS，AGPL 的网络 copyleft 正好命中。因此：
+
+- 允许：借鉴设计决策（如整条流水线单事务 + 末步落 processed 水位的 at-least-once 幂等）、参考反面经验（`migrations/008_drop_hypotheses.sql` 记录了 hypotheses 表建后即废，被 `layer='suspected'/'confirmed'` + supersedes 取代）。
+- 禁止：把 `agent/sleep/*.py`（含 `_maturity.py` 的分层衰减表、`disputes.py` 的争议解决）逐行翻译进 `memory-v2-lifecycle.mjs`。取得商业授权前不做。
+
+Mi-Memory 仓库本身**没有代码**，只有论文 PDF、项目页和图；MemStack / MemSense / D²ACCI / E²MEND / LiteMem 均无参考实现。唯一可直接使用的产物就是 MemFuseBench 数据集。
+
+## 4. 用法
+
+```bash
+# 全量 357 题，约 3.5s
+npm run bench:memory-v2-memfuse
+
+# 单场景 / 单维度 / 冒烟
+node scripts/run-memory-v2-memfuse-bench.mjs --scenario sc1 --limit 20
+node scripts/run-memory-v2-memfuse-bench.mjs --dimension multi_source_conflict_arbitration
+node scripts/run-memory-v2-memfuse-bench.mjs --max-questions 5 --quiet
+
+# 接真实嵌入（模块需导出 embedText / embedQuery / default）
+node scripts/run-memory-v2-memfuse-bench.mjs --embedding-module ./scripts/my-embedder.mjs
+
+# 事件文本前置 "[device · location]" 来源标签
+node scripts/run-memory-v2-memfuse-bench.mjs --source-tags
+
+# 落 JSON 报告
+node scripts/run-memory-v2-memfuse-bench.mjs --out .release-gate/memfuse.json
+```
+
+`--source-tags` 实测把候选召回从 36.6% 抬到 38.2%，但最终召回不变（13.5%），说明当前排序器吃不到 provenance 前缀带来的信息。
+
+单测（不依赖外部数据集）：
+
+```bash
+npm run verify:memory-v2-memfuse-bench
+```
+
+## 5. 指标定义
+
+- `candidateRecall` — 证据事件进入候选池（向量 top-N ∪ 最近 N ∪ keyword 匹配）的比例。**这是排序无关的天花板。**
+- `recall@k` — 证据事件出现在最终 top-k 的比例。
+- `rankingLoss` = `candidateRecall - recall@k`。大于 0 表示证据已被检索到但被排序丢弃。**这是决定"该修检索还是该修排序"的关键拆分**，对应 Mi-Memory 讲的 diagnostic trace。
+- `hitAny@k` — 至少召回一条证据的问题占比。
+- `checklistCoverage` — `answer_checklist` 中每个 point 至少召回一条其 `source_events` 的比例。这是最接近 MemFuse 论文自身评分（answer-checklist coverage）的检索侧代理，无需 LLM 评审。
+- `distractorRate` — top-k 中 `source` 为 `noise` / `adversarial` 的占比。这些事件按数据集构造永远不属于任何答案的证据集。
+
+## 6. 这份基线解锁了什么
+
+`memory-v2-lifecycle.mjs` 的 `promote` / `compact` / `reflect` / decay 目前 flag 全关、`reflect()` 恒返回 `updated: 0`。在没有基线之前，往里面填任何算法都无法验证对错。现在有了一个 3.5 秒可重跑、357 题、能把候选损失和排序损失分开的度量，这个前置条件解除。
+
+按 rankingLoss 的量级，优先级排序是：先修排序（`rankHybridCandidates` 的主键选择、`fetchKeywordCandidates` 的排序方式），再谈候选生成和 consolidation 算法。这两处都是几十行的局部改动，而且每一步都能用本文的表格直接对比。
+
+## 7. 相关文件
+
+- `memory-v2-memfuse-bench.mjs` — 数据集加载、语料/案例构建、评测与聚合
+- `memory-v2-memfuse-bench.test.mjs` — 内联 fixture，19 个用例，不依赖外部数据集
+- `scripts/run-memory-v2-memfuse-bench.mjs` — CLI
+- `memory-v2-pgvector.mjs` — 被测的生产排序路径（`rankHybridCandidates` / `fetchKeywordCandidates` / `extractKeywordTerms`）
+- `memory-v2-recall-benchmark.mjs` — 既有的 4 例中文手写基线，与本文互补

memory-v2-memfuse-bench.mjs

@@ -0,0 +1,600 @@
+import os from 'node:os';
+import path from 'node:path';
+import { readFile as fsReadFile } from 'node:fs/promises';
+
+import { createPgvectorMemoryBackend } from './memory-v2-pgvector.mjs';
+
+/**
+ * MemFuseBench retrieval harness for Memory V2.
+ *
+ * MemFuseBench (Mi-Memory / Darwin Agent Team, MIT) ships 357 evidence-grounded
+ * questions over 7,823 source-tagged events. This module turns those scenarios
+ * into a corpus + case list and runs them through the *production* pgvector
+ * backend so the measured numbers reflect `rankHybridCandidates` and
+ * `extractKeywordTerms`, not a parallel scoring implementation.
+ *
+ * The dataset itself is never vendored into this repository. It is read from an
+ * external checkout and every entry point degrades to
+ * `{ available: false, reason }` when the file is absent, so CI stays green
+ * without it.
+ */
+
+export const MEMFUSE_DATASET_ENV = 'MEMFUSE_BENCH_DATASET';
+
+export const MEMFUSE_DIMENSIONS = Object.freeze([
+  'cross_device_causal_reasoning',
+  'cross_device_information_fusion',
+  'multi_source_conflict_arbitration',
+  'cross_user_information_synthesis',
+  'cross_user_query',
+  'perspective_difference',
+]);
+
+// Events tagged with these sources are adversarial by construction: they look
+// topical but are never part of any answer's evidence set.
+export const MEMFUSE_DISTRACTOR_SOURCES = Object.freeze(['noise', 'adversarial']);
+
+const DEFAULT_RELATIVE_DATASET_PATH = path.join(
+  'mi-memory',
+  'MemFuse',
+  'MemFuseBench',
+  'memfusebench_dataset.json',
+);
+
+export function resolveMemFuseDatasetPath(env = process.env) {
+  const explicit = String(env?.[MEMFUSE_DATASET_ENV] ?? '').trim();
+  if (explicit) return path.resolve(explicit);
+  const projectRoot = String(env?.MEMFUSE_PROJECT_ROOT ?? '').trim();
+  const base = projectRoot || path.join(os.homedir(), 'Project');
+  return path.join(base, DEFAULT_RELATIVE_DATASET_PATH);
+}
+
+function asArray(value) {
+  return Array.isArray(value) ? value : [];
+}
+
+function nonEmptyString(value) {
+  const text = String(value ?? '').trim();
+  return text.length > 0 ? text : null;
+}
+
+export function normalizeMemFuseDataset(raw) {
+  const scenarios = asArray(raw?.scenarios)
+    .map((scenario) => {
+      const scenarioId = nonEmptyString(scenario?.scenario_id);
+      if (!scenarioId) return null;
+      const episodes = asArray(scenario?.episodes)
+        .map((episode) => {
+          const episodeId = nonEmptyString(episode?.episode_id);
+          const events = asArray(episode?.events)
+            .map((event) => {
+              const eventId = nonEmptyString(event?.event_id);
+              const description = nonEmptyString(event?.description);
+              if (!eventId || !description) return null;
+              return {
+                eventId,
+                description,
+                device: nonEmptyString(event?.device),
+                modality: nonEmptyString(event?.modality) ?? 'event',
+                location: nonEmptyString(event?.location),
+                source: nonEmptyString(event?.source) ?? 'unknown',
+                timestamp: nonEmptyString(event?.timestamp),
+                characters: asArray(event?.characters).map(String),
+              };
+            })
+            .filter(Boolean);
+          if (!episodeId || events.length === 0) return null;
+          return { episodeId, events };
+        })
+        .filter(Boolean);
+      const questions = asArray(scenario?.questions)
+        .map((question) => {
+          const questionId = nonEmptyString(question?.question_id);
+          const text = nonEmptyString(question?.question);
+          const evidenceEventIds = [
+            ...new Set(asArray(question?.evidence_event_ids).map(String).filter(Boolean)),
+          ];
+          if (!questionId || !text || evidenceEventIds.length === 0) return null;
+          const checklist = asArray(question?.answer_checklist)
+            .map((entry) => ({
+              point: nonEmptyString(entry?.point) ?? '',
+              sourceEvents: [
+                ...new Set(asArray(entry?.source_events).map(String).filter(Boolean)),
+              ],
+            }))
+            .filter((entry) => entry.sourceEvents.length > 0);
+          return {
+            questionId,
+            scenarioId,
+            question: text,
+            evidenceEventIds,
+            checklist,
+            dimension: nonEmptyString(question?.dimension) ?? 'unknown',
+            questionUser: nonEmptyString(question?.question_user),
+            questionTime: nonEmptyString(question?.question_time),
+            questionDevice: nonEmptyString(question?.question_device),
+          };
+        })
+        .filter(Boolean);
+      if (episodes.length === 0 || questions.length === 0) return null;
+      return {
+        scenarioId,
+        description: nonEmptyString(scenario?.description),
+        timeSpan: nonEmptyString(scenario?.time_span),
+        episodes,
+        questions,
+      };
+    })
+    .filter(Boolean);
+
+  if (scenarios.length === 0) {
+    return { ok: false, reason: 'dataset_has_no_usable_scenarios' };
+  }
+
+  const eventCount = scenarios.reduce(
+    (total, scenario) =>
+      total + scenario.episodes.reduce((sum, episode) => sum + episode.events.length, 0),
+    0,
+  );
+  const questionCount = scenarios.reduce(
+    (total, scenario) => total + scenario.questions.length,
+    0,
+  );
+
+  return {
+    ok: true,
+    dataset: {
+      scenarios,
+      stats: { scenarioCount: scenarios.length, eventCount, questionCount },
+    },
+  };
+}
+
+export async function loadMemFuseDataset({
+  datasetPath = null,
+  env = process.env,
+  readFile = fsReadFile,
+} = {}) {
+  const resolvedPath = datasetPath ? path.resolve(datasetPath) : resolveMemFuseDatasetPath(env);
+  let payload;
+  try {
+    payload = await readFile(resolvedPath, 'utf8');
+  } catch (err) {
+    return {
+      available: false,
+      path: resolvedPath,
+      reason: err?.code === 'ENOENT' ? 'dataset_not_found' : 'dataset_read_failed',
+      error: err instanceof Error ? err.message : String(err),
+    };
+  }
+  let raw;
+  try {
+    raw = JSON.parse(payload);
+  } catch (err) {
+    return {
+      available: false,
+      path: resolvedPath,
+      reason: 'dataset_parse_failed',
+      error: err instanceof Error ? err.message : String(err),
+    };
+  }
+  const normalized = normalizeMemFuseDataset(raw);
+  if (!normalized.ok) {
+    return { available: false, path: resolvedPath, reason: normalized.reason };
+  }
+  return { available: true, path: resolvedPath, ...normalized.dataset };
+}
+
+/**
+ * Flatten a scenario into rows shaped like the `memory_embeddings` table so the
+ * pgvector backend can consume them unchanged.
+ *
+ * `includeSourceTags` prepends `[device · location]`. It defaults to false so the
+ * baseline measures retrieval over raw event text rather than over a prefix
+ * format chosen here.
+ */
+export function buildScenarioCorpus(scenario, { includeSourceTags = false } = {}) {
+  const rows = [];
+  const byId = new Map();
+  for (const episode of asArray(scenario?.episodes)) {
+    for (const event of asArray(episode?.events)) {
+      const tags = [event.device, event.location].filter(Boolean).join(' · ');
+      const content =
+        includeSourceTags && tags ? `[${tags}] ${event.description}` : event.description;
+      const timestampMs = event.timestamp ? Date.parse(event.timestamp) : NaN;
+      const isoTimestamp = Number.isFinite(timestampMs)
+        ? new Date(timestampMs).toISOString()
+        : null;
+      const row = {
+        id: event.eventId,
+        content,
+        type: event.modality,
+        created_at: isoTimestamp,
+        updated_at: isoTimestamp,
+        episodeId: episode.episodeId,
+        source: event.source,
+        device: event.device,
+        location: event.location,
+        timestampMs: Number.isFinite(timestampMs) ? timestampMs : 0,
+      };
+      rows.push(row);
+      byId.set(row.id, row);
+    }
+  }
+  return { rows, byId };
+}
+
+export function buildScenarioCases(
+  scenario,
+  { dimensions = null, maxQuestions = null, questionIds = null } = {},
+) {
+  const dimensionFilter = dimensions?.length ? new Set(dimensions) : null;
+  const idFilter = questionIds?.length ? new Set(questionIds) : null;
+  const cases = asArray(scenario?.questions).filter((question) => {
+    if (dimensionFilter && !dimensionFilter.has(question.dimension)) return false;
+    if (idFilter && !idFilter.has(question.questionId)) return false;
+    return true;
+  });
+  const capped = Number(maxQuestions);
+  return Number.isFinite(capped) && capped > 0 ? cases.slice(0, capped) : cases;
+}
+
+const CJK_RANGE = /[\u4e00-\u9fff]/u;
+
+function tokenizeForEmbedding(text) {
+  const normalized = String(text ?? '')
+    .normalize('NFKC')
+    .toLowerCase();
+  const tokens = [];
+  for (const match of normalized.matchAll(/[a-z0-9]{2,}/g)) tokens.push(match[0]);
+  if (CJK_RANGE.test(normalized)) {
+    const cjk = normalized.replace(/[^\u4e00-\u9fff]/gu, '');
+    for (let index = 0; index + 2 <= cjk.length; index += 1) {
+      tokens.push(cjk.slice(index, index + 2));
+    }
+  }
+  return tokens;
+}
+
+function hashToken(token, dimensions) {
+  let hash = 2166136261;
+  for (let index = 0; index < token.length; index += 1) {
+    hash ^= token.charCodeAt(index);
+    hash = Math.imul(hash, 16777619);
+  }
+  return Math.abs(hash) % dimensions;
+}
+
+/**
+ * Deterministic offline stand-in for a sentence embedder: hashed bag-of-tokens,
+ * L2 normalized. Cosine over these vectors approximates lexical overlap, NOT
+ * semantic similarity, so reports built on it are labelled `lexical-hash`. Pass
+ * a real `embedText` to measure semantic recall.
+ */
+export function createLexicalHashEmbedder({ dimensions = 256 } = {}) {
+  const size = Math.max(16, Math.min(4096, Number(dimensions) || 256));
+  return function embedText(text) {
+    const vector = new Array(size).fill(0);
+    const tokens = tokenizeForEmbedding(text);
+    if (tokens.length === 0) return vector;
+    for (const token of tokens) {
+      vector[hashToken(token, size)] += 1;
+    }
+    let norm = 0;
+    for (const value of vector) norm += value * value;
+    norm = Math.sqrt(norm);
+    if (norm === 0) return vector;
+    for (let index = 0; index < size; index += 1) vector[index] /= norm;
+    return vector;
+  };
+}
+
+function cosineSimilarity(left, right) {
+  const length = Math.min(left.length, right.length);
+  let dot = 0;
+  let leftNorm = 0;
+  let rightNorm = 0;
+  for (let index = 0; index < length; index += 1) {
+    dot += left[index] * right[index];
+    leftNorm += left[index] * left[index];
+    rightNorm += right[index] * right[index];
+  }
+  if (leftNorm === 0 || rightNorm === 0) return 0;
+  return dot / (Math.sqrt(leftNorm) * Math.sqrt(rightNorm));
+}
+
+function parseVectorLiteral(literal) {
+  const text = String(literal ?? '').trim();
+  if (!text.startsWith('[') || !text.endsWith(']')) return null;
+  const inner = text.slice(1, -1);
+  if (!inner) return [];
+  const parts = inner.split(',').map((part) => Number(part));
+  return parts.some((part) => !Number.isFinite(part)) ? null : parts;
+}
+
+function likeParamToTerm(param) {
+  return String(param ?? '')
+    .replace(/^%+/, '')
+    .replace(/%+$/, '')
+    .toLowerCase();
+}
+
+/**
+ * Fake pg pool that emulates the two query shapes issued by
+ * `createPgvectorMemoryBackend.resolve`: the vector/recent candidate union and
+ * the `content ILIKE` keyword fallback. Emulating the SQL rather than bypassing
+ * it keeps the production ranking path under test.
+ */
+export function createCorpusPool({ rows, embedText, embeddingCache = new Map() }) {
+  const corpus = asArray(rows);
+  const byRecency = [...corpus].sort((left, right) => right.timestampMs - left.timestampMs);
+
+  function embeddingFor(row) {
+    if (!embeddingCache.has(row.id)) {
+      embeddingCache.set(row.id, embedText(row.content));
+    }
+    return embeddingCache.get(row.id);
+  }
+
+  function toResultRow(row, score) {
+    return {
+      id: row.id,
+      content: row.content,
+      type: row.type,
+      created_at: row.created_at,
+      updated_at: row.updated_at,
+      score,
+    };
+  }
+
+  let vectorQueryCount = 0;
+  let keywordQueryCount = 0;
+  // Every id handed back to the backend, i.e. the candidate pool that ranking
+  // then narrows to top-k. Tracking it separates "candidate generation lost the
+  // evidence" from "ranking lost the evidence".
+  const returnedIds = new Set();
+
+  function trackRows(rows) {
+    for (const row of rows) returnedIds.add(row.id);
+    return rows;
+  }
+
+  return {
+    stats: () => ({
+      vectorQueryCount,
+      keywordQueryCount,
+      corpusSize: corpus.length,
+      candidateIds: new Set(returnedIds),
+    }),
+    async query(sql, params = []) {
+      const text = String(sql ?? '');
+      if (text.includes('vector_candidates')) {
+        vectorQueryCount += 1;
+        const queryVector = parseVectorLiteral(params[1]);
+        const limit = Math.max(1, Number(params[2]) || 50);
+        const scored = corpus.map((row) => ({
+          row,
+          score: queryVector ? cosineSimilarity(queryVector, embeddingFor(row)) : 0,
+        }));
+        const scoreById = new Map(scored.map((entry) => [entry.row.id, entry.score]));
+        const vectorTop = [...scored]
+          .sort((left, right) => right.score - left.score)
+          .slice(0, limit)
+          .map((entry) => entry.row);
+        const recentTop = byRecency.slice(0, limit);
+        const merged = new Map();
+        for (const row of [...vectorTop, ...recentTop]) {
+          if (!merged.has(row.id)) merged.set(row.id, row);
+        }
+        return {
+          rows: trackRows(
+            [...merged.values()].map((row) => toResultRow(row, scoreById.get(row.id) ?? 0)),
+          ),
+        };
+      }
+      if (text.includes('ILIKE')) {
+        keywordQueryCount += 1;
+        const limit = Math.max(1, Number(params[params.length - 1]) || 20);
+        const terms = params
+          .slice(1, params.length - 1)
+          .map(likeParamToTerm)
+          .filter(Boolean);
+        if (terms.length === 0) return { rows: [] };
+        const matched = byRecency.filter((row) => {
+          const haystack = row.content.toLowerCase();
+          return terms.some((term) => haystack.includes(term));
+        });
+        return { rows: trackRows(matched.slice(0, limit).map((row) => toResultRow(row, 1))) };
+      }
+      return { rows: [] };
+    },
+  };
+}
+
+function checklistCoverage(checklist, retrievedIds) {
+  if (!checklist?.length) return null;
+  let covered = 0;
+  for (const entry of checklist) {
+    if (entry.sourceEvents.some((eventId) => retrievedIds.has(eventId))) covered += 1;
+  }
+  return covered / checklist.length;
+}
+
+export async function runMemFuseBenchCase({
+  testCase,
+  corpus,
+  embedText,
+  limit = 20,
+  candidateLimit = 100,
+  embeddingCache = new Map(),
+}) {
+  const pool = createCorpusPool({ rows: corpus.rows, embedText, embeddingCache });
+  const backend = createPgvectorMemoryBackend({ enabled: true, embedQuery: embedText, pool });
+  const result = await backend.resolve({
+    userId: 'memfuse-bench-user',
+    query: testCase.question,
+    limit,
+    candidateLimit,
+  });
+
+  const retrieved = result.memories.map((memory) => memory.id).filter(Boolean);
+  const retrievedIds = new Set(retrieved);
+  const gold = new Set(testCase.evidenceEventIds);
+  const hits = retrieved.filter((id) => gold.has(id));
+  const firstHitIndex = retrieved.findIndex((id) => gold.has(id));
+  const distractors = retrieved.filter((id) =>
+    MEMFUSE_DISTRACTOR_SOURCES.includes(corpus.byId.get(id)?.source),
+  );
+
+  const { candidateIds } = pool.stats();
+  const candidateHits = [...gold].filter((id) => candidateIds.has(id));
+  const candidateRecall = gold.size > 0 ? candidateHits.length / gold.size : 0;
+  const recall = gold.size > 0 ? hits.length / gold.size : 0;
+
+  return {
+    questionId: testCase.questionId,
+    scenarioId: testCase.scenarioId,
+    dimension: testCase.dimension,
+    goldCount: gold.size,
+    returned: retrieved.length,
+    hitCount: hits.length,
+    recall,
+    precision: retrieved.length > 0 ? hits.length / retrieved.length : 0,
+    hitAny: hits.length > 0,
+    candidateCount: candidateIds.size,
+    candidateHitCount: candidateHits.length,
+    candidateRecall,
+    // Positive value = evidence reached the candidate pool but ranking dropped
+    // it. This is the split that decides whether to fix retrieval or ranking.
+    rankingLoss: Math.max(0, candidateRecall - recall),
+    firstHitRank: firstHitIndex >= 0 ? firstHitIndex + 1 : null,
+    reciprocalRank: firstHitIndex >= 0 ? 1 / (firstHitIndex + 1) : 0,
+    checklistCoverage: checklistCoverage(testCase.checklist, retrievedIds),
+    distractorCount: distractors.length,
+  };
+}
+
+function mean(values) {
+  const usable = values.filter((value) => Number.isFinite(value));
+  if (usable.length === 0) return 0;
+  return usable.reduce((total, value) => total + value, 0) / usable.length;
+}
+
+function aggregate(results) {
+  const coverage = results
+    .map((item) => item.checklistCoverage)
+    .filter((value) => Number.isFinite(value));
+  return {
+    caseCount: results.length,
+    recallAtK: mean(results.map((item) => item.recall)),
+    precisionAtK: mean(results.map((item) => item.precision)),
+    candidateRecall: mean(results.map((item) => item.candidateRecall)),
+    rankingLoss: mean(results.map((item) => item.rankingLoss)),
+    hitAnyRate: results.length
+      ? results.filter((item) => item.hitAny).length / results.length
+      : 0,
+    checklistCoverage: coverage.length ? mean(coverage) : null,
+    mrr: mean(results.map((item) => item.reciprocalRank)),
+    distractorRate: results.length
+      ? mean(results.map((item) => (item.returned ? item.distractorCount / item.returned : 0)))
+      : 0,
+  };
+}
+
+export async function runMemFuseBench({
+  dataset,
+  scenarioIds = null,
+  dimensions = null,
+  maxQuestionsPerScenario = null,
+  questionIds = null,
+  limit = 20,
+  candidateLimit = 100,
+  embedText = null,
+  includeSourceTags = false,
+  onProgress = null,
+} = {}) {
+  const scenarios = asArray(dataset?.scenarios);
+  if (scenarios.length === 0) throw new Error('runMemFuseBench requires a loaded dataset');
+  const embedder = typeof embedText === 'function' ? embedText : createLexicalHashEmbedder();
+  const embeddingMode = typeof embedText === 'function' ? 'external' : 'lexical-hash';
+  const scenarioFilter = scenarioIds?.length ? new Set(scenarioIds) : null;
+  // The pgvector backend clamps its own limit to 50; mirror it so the reported
+  // `limit` never overstates how many rows were actually considered.
+  const effectiveLimit = Math.max(1, Math.min(50, Number(limit) || 20));
+
+  const results = [];
+  const scenarioReports = [];
+  for (const scenario of scenarios) {
+    if (scenarioFilter && !scenarioFilter.has(scenario.scenarioId)) continue;
+    const cases = buildScenarioCases(scenario, {
+      dimensions,
+      maxQuestions: maxQuestionsPerScenario,
+      questionIds,
+    });
+    if (cases.length === 0) continue;
+    const corpus = buildScenarioCorpus(scenario, { includeSourceTags });
+    // One cache per scenario: corpus embeddings are reused across that
+    // scenario's questions, which is where nearly all the cost sits.
+    const embeddingCache = new Map();
+    const scenarioResults = [];
+    for (const testCase of cases) {
+      const caseResult = await runMemFuseBenchCase({
+        testCase,
+        corpus,
+        embedText: embedder,
+        limit: effectiveLimit,
+        candidateLimit,
+        embeddingCache,
+      });
+      scenarioResults.push(caseResult);
+      results.push(caseResult);
+      if (typeof onProgress === 'function') {
+        onProgress({ scenarioId: scenario.scenarioId, completed: results.length, ...caseResult });
+      }
+    }
+    scenarioReports.push({
+      scenarioId: scenario.scenarioId,
+      corpusSize: corpus.rows.length,
+      ...aggregate(scenarioResults),
+    });
+  }
+
+  if (results.length === 0) throw new Error('runMemFuseBench selected zero questions');
+
+  const byDimension = {};
+  for (const dimension of new Set(results.map((item) => item.dimension))) {
+    byDimension[dimension] = aggregate(results.filter((item) => item.dimension === dimension));
+  }
+
+  return {
+    limit: effectiveLimit,
+    candidateLimit,
+    embeddingMode,
+    includeSourceTags,
+    overall: aggregate(results),
+    byScenario: scenarioReports,
+    byDimension,
+    results,
+  };
+}
+
+export function summarizeMemFuseBench(report) {
+  const round = (value) => (Number.isFinite(value) ? Number(value.toFixed(4)) : null);
+  return {
+    limit: report.limit,
+    embeddingMode: report.embeddingMode,
+    caseCount: report.overall.caseCount,
+    recallAtK: round(report.overall.recallAtK),
+    precisionAtK: round(report.overall.precisionAtK),
+    candidateRecall: round(report.overall.candidateRecall),
+    rankingLoss: round(report.overall.rankingLoss),
+    hitAnyRate: round(report.overall.hitAnyRate),
+    checklistCoverage: round(report.overall.checklistCoverage),
+    mrr: round(report.overall.mrr),
+    distractorRate: round(report.overall.distractorRate),
+    weakestDimension:
+      Object.entries(report.byDimension)
+        .sort((left, right) => left[1].recallAtK - right[1].recallAtK)
+        .map(([dimension]) => dimension)[0] ?? null,
+  };
+}

memory-v2-memfuse-bench.test.mjs

@@ -0,0 +1,449 @@
+import assert from 'node:assert/strict';
+import test from 'node:test';
+
+import {
+  MEMFUSE_DATASET_ENV,
+  buildScenarioCases,
+  buildScenarioCorpus,
+  createCorpusPool,
+  createLexicalHashEmbedder,
+  loadMemFuseDataset,
+  normalizeMemFuseDataset,
+  resolveMemFuseDatasetPath,
+  runMemFuseBench,
+  runMemFuseBenchCase,
+  summarizeMemFuseBench,
+} from './memory-v2-memfuse-bench.mjs';
+
+// Mirrors the real MemFuseBench shape (scenarios → episodes → events, plus
+// questions carrying evidence_event_ids and answer_checklist) at a size that
+// keeps this suite hermetic: the 6MB upstream dataset is never required.
+function makeRawDataset() {
+  return {
+    metadata: { total_questions: 3 },
+    scenarios: [
+      {
+        scenario_id: 'sc1',
+        description: 'Scenario sc1',
+        time_span: '2026-05-11 ~ 2026-05-12',
+        episodes: [
+          {
+            episode_id: 'sc1_ep1',
+            events: [
+              {
+                event_id: 'sc1_ep1_e1',
+                device: 'thermostat_living',
+                modality: 'environment',
+                characters: [],
+                description: 'Living room thermostat reading: temperature 21.8C, humidity 47%.',
+                timestamp: '2026-05-11T06:00:00',
+                location: 'living_room',
+                source: 'periodic',
+              },
+              {
+                event_id: 'sc1_ep1_e2',
+                device: 'purifier_living',
+                modality: 'appliance',
+                characters: ['Sarah'],
+                description: 'Sarah switched the air purifier to turbo because of pollen.',
+                timestamp: '2026-05-11T08:12:00',
+                location: 'living_room',
+                source: 'storyline',
+              },
+              {
+                event_id: 'sc1_ep1_e3',
+                device: 'curtain_living',
+                modality: 'appliance',
+                characters: ['Sarah'],
+                description: 'Smart curtains closed to reduce pollen entering the living room.',
+                timestamp: '2026-05-11T08:15:00',
+                location: 'living_room',
+                source: 'storyline',
+              },
+              {
+                event_id: 'sc1_ep1_e4',
+                device: 'phone_david',
+                modality: 'message',
+                characters: ['David'],
+                description: 'Unrelated chatter about a weekend hiking trip to the coast.',
+                timestamp: '2026-05-11T09:00:00',
+                location: 'office',
+                source: 'noise',
+              },
+              {
+                event_id: 'sc1_ep1_e5',
+                device: 'speaker_kitchen',
+                modality: 'audio',
+                characters: ['Ethan'],
+                description: 'Adversarial lookalike: a podcast episode discussing pollen forecasts.',
+                timestamp: '2026-05-11T09:30:00',
+                location: 'kitchen',
+                source: 'adversarial',
+              },
+            ],
+          },
+        ],
+        questions: [
+          {
+            question_id: 'q1',
+            question: 'Why did Sarah change the purifier and curtains for pollen?',
+            answer: 'Because of pollen.',
+            answer_checklist: [
+              { point: 'purifier to turbo', source_events: ['sc1_ep1_e2'] },
+              { point: 'curtains closed', source_events: ['sc1_ep1_e3'] },
+            ],
+            evidence_event_ids: ['sc1_ep1_e2', 'sc1_ep1_e3'],
+            dimension: 'cross_device_causal_reasoning',
+            question_user: 'David',
+            question_time: '2026-05-11T12:30:00',
+            question_device: 'phone_david',
+          },
+          {
+            question_id: 'q2',
+            question: 'What was the living room thermostat temperature reading?',
+            answer: '21.8C',
+            answer_checklist: [{ point: 'temperature', source_events: ['sc1_ep1_e1'] }],
+            evidence_event_ids: ['sc1_ep1_e1'],
+            dimension: 'cross_device_information_fusion',
+            question_user: 'Sarah',
+            question_time: '2026-05-11T13:00:00',
+            question_device: 'tablet_home',
+          },
+        ],
+      },
+      {
+        scenario_id: 'sc2',
+        episodes: [
+          {
+            episode_id: 'sc2_ep1',
+            events: [
+              {
+                event_id: 'sc2_ep1_e1',
+                device: 'watch_ethan',
+                modality: 'wearable',
+                characters: ['Ethan'],
+                description: 'Wearable recorded an elevated heart rate during the afternoon run.',
+                timestamp: '2026-05-12T17:05:00',
+                location: 'park',
+                source: 'storyline',
+              },
+            ],
+          },
+        ],
+        questions: [
+          {
+            question_id: 'q1',
+            question: 'What did the wearable record during the afternoon run?',
+            answer: 'Elevated heart rate.',
+            answer_checklist: [{ point: 'elevated heart rate', source_events: ['sc2_ep1_e1'] }],
+            evidence_event_ids: ['sc2_ep1_e1'],
+            dimension: 'perspective_difference',
+            question_user: 'Sarah',
+            question_time: '2026-05-12T19:00:00',
+            question_device: 'phone_sarah',
+          },
+        ],
+      },
+    ],
+  };
+}
+
+function loadFixtureDataset() {
+  const normalized = normalizeMemFuseDataset(makeRawDataset());
+  assert.equal(normalized.ok, true);
+  return normalized.dataset;
+}
+
+test('resolveMemFuseDatasetPath prefers the explicit env override', () => {
+  const explicit = resolveMemFuseDatasetPath({ [MEMFUSE_DATASET_ENV]: '/tmp/custom/bench.json' });
+  assert.equal(explicit, '/tmp/custom/bench.json');
+
+  const viaRoot = resolveMemFuseDatasetPath({ MEMFUSE_PROJECT_ROOT: '/srv/refs' });
+  assert.equal(viaRoot, '/srv/refs/mi-memory/MemFuse/MemFuseBench/memfusebench_dataset.json');
+});
+
+test('normalizeMemFuseDataset keeps usable scenarios and reports stats', () => {
+  const normalized = normalizeMemFuseDataset(makeRawDataset());
+  assert.equal(normalized.ok, true);
+  assert.deepEqual(normalized.dataset.stats, {
+    scenarioCount: 2,
+    eventCount: 6,
+    questionCount: 3,
+  });
+  const [scenario] = normalized.dataset.scenarios;
+  assert.equal(scenario.scenarioId, 'sc1');
+  assert.equal(scenario.questions[0].evidenceEventIds.length, 2);
+  assert.equal(scenario.questions[0].checklist.length, 2);
+});
+
+test('normalizeMemFuseDataset drops events and questions that cannot be scored', () => {
+  const raw = makeRawDataset();
+  raw.scenarios[0].episodes[0].events.push({ event_id: 'no_description' });
+  raw.scenarios[0].questions.push({
+    question_id: 'q_no_evidence',
+    question: 'Unscoreable question',
+    evidence_event_ids: [],
+  });
+  const normalized = normalizeMemFuseDataset(raw);
+  assert.equal(normalized.ok, true);
+  assert.equal(normalized.dataset.stats.eventCount, 6);
+  assert.equal(normalized.dataset.stats.questionCount, 3);
+});
+
+test('normalizeMemFuseDataset rejects payloads with no usable scenario', () => {
+  assert.equal(normalizeMemFuseDataset({}).reason, 'dataset_has_no_usable_scenarios');
+  assert.equal(normalizeMemFuseDataset({ scenarios: [{}] }).reason, 'dataset_has_no_usable_scenarios');
+});
+
+test('loadMemFuseDataset degrades instead of throwing when the dataset is absent', async () => {
+  const missing = await loadMemFuseDataset({
+    datasetPath: '/tmp/definitely-absent.json',
+    readFile: async () => {
+      const err = new Error('ENOENT');
+      err.code = 'ENOENT';
+      throw err;
+    },
+  });
+  assert.equal(missing.available, false);
+  assert.equal(missing.reason, 'dataset_not_found');
+
+  const unparsable = await loadMemFuseDataset({
+    datasetPath: '/tmp/broken.json',
+    readFile: async () => 'not json',
+  });
+  assert.equal(unparsable.available, false);
+  assert.equal(unparsable.reason, 'dataset_parse_failed');
+});
+
+test('loadMemFuseDataset returns normalized scenarios on success', async () => {
+  const loaded = await loadMemFuseDataset({
+    datasetPath: '/tmp/fixture.json',
+    readFile: async () => JSON.stringify(makeRawDataset()),
+  });
+  assert.equal(loaded.available, true);
+  assert.equal(loaded.path, '/tmp/fixture.json');
+  assert.equal(loaded.stats.questionCount, 3);
+});
+
+test('buildScenarioCorpus emits pgvector-shaped rows and can prepend source tags', () => {
+  const [scenario] = loadFixtureDataset().scenarios;
+  const plain = buildScenarioCorpus(scenario);
+  assert.equal(plain.rows.length, 5);
+  const first = plain.byId.get('sc1_ep1_e1');
+  assert.equal(first.content, 'Living room thermostat reading: temperature 21.8C, humidity 47%.');
+  assert.equal(first.type, 'environment');
+  assert.equal(first.source, 'periodic');
+  assert.equal(first.episodeId, 'sc1_ep1');
+  assert.equal(first.created_at, new Date('2026-05-11T06:00:00').toISOString());
+
+  const tagged = buildScenarioCorpus(scenario, { includeSourceTags: true });
+  assert.equal(
+    tagged.byId.get('sc1_ep1_e1').content,
+    '[thermostat_living · living_room] Living room thermostat reading: temperature 21.8C, humidity 47%.',
+  );
+});
+
+test('buildScenarioCases filters by dimension, id and cap', () => {
+  const [scenario] = loadFixtureDataset().scenarios;
+  assert.equal(buildScenarioCases(scenario).length, 2);
+  assert.deepEqual(
+    buildScenarioCases(scenario, { dimensions: ['cross_device_causal_reasoning'] }).map(
+      (item) => item.questionId,
+    ),
+    ['q1'],
+  );
+  assert.deepEqual(
+    buildScenarioCases(scenario, { questionIds: ['q2'] }).map((item) => item.questionId),
+    ['q2'],
+  );
+  assert.equal(buildScenarioCases(scenario, { maxQuestions: 1 }).length, 1);
+});
+
+test('createLexicalHashEmbedder is deterministic and L2 normalized', () => {
+  const embed = createLexicalHashEmbedder({ dimensions: 64 });
+  const left = embed('pollen purifier curtains');
+  const right = embed('pollen purifier curtains');
+  assert.deepEqual(left, right);
+  assert.equal(left.length, 64);
+  const norm = Math.sqrt(left.reduce((total, value) => total + value * value, 0));
+  assert.ok(Math.abs(norm - 1) < 1e-9);
+  assert.notDeepEqual(embed('completely different text'), left);
+  assert.deepEqual(
+    embed(''),
+    new Array(64).fill(0),
+  );
+});
+
+test('createCorpusPool emulates the vector union and the ILIKE fallback', async () => {
+  const [scenario] = loadFixtureDataset().scenarios;
+  const corpus = buildScenarioCorpus(scenario);
+  const embed = createLexicalHashEmbedder({ dimensions: 128 });
+  const pool = createCorpusPool({ rows: corpus.rows, embedText: embed });
+
+  const vectorSql = 'WITH vector_candidates AS (SELECT ...) SELECT DISTINCT ON (id) ...';
+  const vectorResult = await pool.query(vectorSql, [
+    'user',
+    `[${embed('pollen purifier').join(',')}]`,
+    3,
+  ]);
+  assert.ok(vectorResult.rows.length > 0);
+  assert.ok(vectorResult.rows.every((row) => typeof row.score === 'number'));
+  assert.ok(vectorResult.rows.some((row) => row.id === 'sc1_ep1_e2'));
+
+  const keywordResult = await pool.query(
+    'SELECT id, content FROM memory_embeddings WHERE user_id = $1 AND (content ILIKE $2) LIMIT $3',
+    ['user', '%curtains%', 10],
+  );
+  assert.deepEqual(
+    keywordResult.rows.map((row) => row.id),
+    ['sc1_ep1_e3'],
+  );
+
+  const unknown = await pool.query('SELECT 1', []);
+  assert.deepEqual(unknown.rows, []);
+  assert.equal(pool.stats().corpusSize, 5);
+  assert.equal(pool.stats().vectorQueryCount, 1);
+  assert.equal(pool.stats().keywordQueryCount, 1);
+});
+
+test('runMemFuseBenchCase scores recall, checklist coverage and distractors', async () => {
+  const [scenario] = loadFixtureDataset().scenarios;
+  const corpus = buildScenarioCorpus(scenario);
+  const [testCase] = scenario.questions;
+  const result = await runMemFuseBenchCase({
+    testCase,
+    corpus,
+    embedText: createLexicalHashEmbedder({ dimensions: 256 }),
+    limit: 5,
+  });
+
+  assert.equal(result.questionId, 'q1');
+  assert.equal(result.dimension, 'cross_device_causal_reasoning');
+  assert.equal(result.goldCount, 2);
+  assert.equal(result.hitCount, 2);
+  assert.equal(result.recall, 1);
+  assert.equal(result.hitAny, true);
+  assert.equal(result.checklistCoverage, 1);
+  assert.ok(result.firstHitRank >= 1);
+  assert.ok(result.reciprocalRank > 0);
+  assert.ok(result.precision > 0 && result.precision <= 1);
+  // e4 (noise) and e5 (adversarial) are the only planted distractors.
+  assert.ok(result.distractorCount <= 2);
+  // Small corpus: every row reaches the candidate pool, so ranking loses nothing.
+  assert.equal(result.candidateRecall, 1);
+  assert.equal(result.rankingLoss, 0);
+  assert.equal(result.candidateCount, 5);
+});
+
+test('runMemFuseBenchCase separates candidate recall from ranking loss', async () => {
+  const [scenario] = loadFixtureDataset().scenarios;
+  const corpus = buildScenarioCorpus(scenario);
+  const [testCase] = scenario.questions;
+  // limit 1 forces ranking to drop evidence that candidate generation found.
+  const result = await runMemFuseBenchCase({
+    testCase,
+    corpus,
+    embedText: createLexicalHashEmbedder({ dimensions: 256 }),
+    limit: 1,
+  });
+  assert.equal(result.candidateRecall, 1);
+  assert.ok(result.recall < 1);
+  assert.ok(result.rankingLoss > 0);
+  assert.equal(result.rankingLoss, result.candidateRecall - result.recall);
+});
+
+test('runMemFuseBenchCase reports a clean miss when nothing relevant exists', async () => {
+  const [scenario] = loadFixtureDataset().scenarios;
+  const corpus = buildScenarioCorpus(scenario);
+  const result = await runMemFuseBenchCase({
+    testCase: {
+      questionId: 'synthetic-miss',
+      scenarioId: 'sc1',
+      dimension: 'cross_user_query',
+      question: 'zzz nonexistent topic zzz',
+      evidenceEventIds: ['sc1_ep1_missing'],
+      checklist: [{ point: 'absent', sourceEvents: ['sc1_ep1_missing'] }],
+    },
+    corpus,
+    embedText: createLexicalHashEmbedder({ dimensions: 256 }),
+    limit: 5,
+  });
+  assert.equal(result.hitCount, 0);
+  assert.equal(result.recall, 0);
+  assert.equal(result.hitAny, false);
+  assert.equal(result.firstHitRank, null);
+  assert.equal(result.reciprocalRank, 0);
+  assert.equal(result.checklistCoverage, 0);
+});
+
+test('runMemFuseBench aggregates overall, per-scenario and per-dimension metrics', async () => {
+  const dataset = loadFixtureDataset();
+  const report = await runMemFuseBench({ dataset, limit: 5 });
+
+  assert.equal(report.embeddingMode, 'lexical-hash');
+  assert.equal(report.limit, 5);
+  assert.equal(report.overall.caseCount, 3);
+  assert.ok(report.overall.recallAtK > 0);
+  assert.ok(report.overall.hitAnyRate > 0);
+  assert.deepEqual(
+    report.byScenario.map((item) => item.scenarioId),
+    ['sc1', 'sc2'],
+  );
+  assert.equal(report.byScenario[0].corpusSize, 5);
+  assert.ok(Object.keys(report.byDimension).length >= 2);
+  assert.equal(report.results.length, 3);
+});
+
+test('runMemFuseBench honours scenario, dimension and cap filters', async () => {
+  const dataset = loadFixtureDataset();
+  const report = await runMemFuseBench({
+    dataset,
+    scenarioIds: ['sc1'],
+    dimensions: ['cross_device_information_fusion'],
+    limit: 5,
+  });
+  assert.equal(report.overall.caseCount, 1);
+  assert.equal(report.results[0].questionId, 'q2');
+
+  const capped = await runMemFuseBench({ dataset, maxQuestionsPerScenario: 1, limit: 5 });
+  assert.equal(capped.overall.caseCount, 2);
+});
+
+test('runMemFuseBench clamps limit to the pgvector backend ceiling', async () => {
+  const report = await runMemFuseBench({ dataset: loadFixtureDataset(), limit: 500 });
+  assert.equal(report.limit, 50);
+});
+
+test('runMemFuseBench marks an injected embedder as external and reports progress', async () => {
+  const dataset = loadFixtureDataset();
+  const seen = [];
+  const report = await runMemFuseBench({
+    dataset,
+    limit: 5,
+    embedText: createLexicalHashEmbedder({ dimensions: 32 }),
+    onProgress: (event) => seen.push(event.questionId),
+  });
+  assert.equal(report.embeddingMode, 'external');
+  assert.equal(seen.length, 3);
+});
+
+test('runMemFuseBench rejects an empty dataset or an empty selection', async () => {
+  await assert.rejects(() => runMemFuseBench({ dataset: { scenarios: [] } }), /requires a loaded dataset/);
+  await assert.rejects(
+    () => runMemFuseBench({ dataset: loadFixtureDataset(), dimensions: ['not_a_dimension'] }),
+    /selected zero questions/,
+  );
+});
+
+test('summarizeMemFuseBench rounds metrics and names the weakest dimension', async () => {
+  const report = await runMemFuseBench({ dataset: loadFixtureDataset(), limit: 5 });
+  const summary = summarizeMemFuseBench(report);
+  assert.equal(summary.caseCount, 3);
+  assert.equal(summary.embeddingMode, 'lexical-hash');
+  assert.ok(summary.recallAtK >= 0 && summary.recallAtK <= 1);
+  assert.ok(summary.candidateRecall >= summary.recallAtK);
+  assert.ok(summary.rankingLoss >= 0);
+  assert.ok(summary.mrr >= 0 && summary.mrr <= 1);
+  assert.ok(summary.distractorRate >= 0 && summary.distractorRate <= 1);
+  assert.ok(typeof summary.weakestDimension === 'string');
+});

package.json

@@ -69,6 +69,8 @@
     "repair:memory-v2-candidates": "node scripts/repair-memory-v2-candidates.mjs",
     "verify:memory-v2-shadow-audit": "node --test memory-v2-shadow-audit.test.mjs",
     "verify:memory-v2-recall-benchmark": "node --test memory-v2-recall-benchmark.test.mjs",
+    "verify:memory-v2-memfuse-bench": "node --test memory-v2-memfuse-bench.test.mjs",
+    "bench:memory-v2-memfuse": "node scripts/run-memory-v2-memfuse-bench.mjs",
     "trace:mindspace-artifact": "node scripts/trace-mindspace-artifact.mjs",
     "check:conversation-package-manifest": "node scripts/check-conversation-package-manifest.mjs",
     "check:memory-v2-phase-a": "node scripts/check-memory-v2-phase-a-ready.mjs",

scripts/run-memory-v2-memfuse-bench.mjs

@@ -0,0 +1,242 @@
+#!/usr/bin/env node
+/**
+ * Run the MemFuseBench retrieval benchmark against the Memory V2 pgvector
+ * ranking path.
+ *
+ * The dataset lives outside this repository (MIT, Mi-Memory / Darwin Agent
+ * Team). Point at it with --dataset or MEMFUSE_BENCH_DATASET; without it the
+ * run reports `skipped` and exits 0 unless --strict is passed.
+ *
+ * Usage:
+ *   node scripts/run-memory-v2-memfuse-bench.mjs
+ *   node scripts/run-memory-v2-memfuse-bench.mjs --scenario sc1 --limit 20
+ *   node scripts/run-memory-v2-memfuse-bench.mjs --dimension multi_source_conflict_arbitration
+ *   node scripts/run-memory-v2-memfuse-bench.mjs --embedding-module ./scripts/my-embedder.mjs
+ *   node scripts/run-memory-v2-memfuse-bench.mjs --json --out .release-gate/memfuse.json
+ */
+
+import fs from 'node:fs/promises';
+import path from 'node:path';
+
+import {
+  MEMFUSE_DIMENSIONS,
+  loadMemFuseDataset,
+  runMemFuseBench,
+  summarizeMemFuseBench,
+} from '../memory-v2-memfuse-bench.mjs';
+import { resolveEmbeddingModuleSpecifier } from '../memory-v2-recall-benchmark.mjs';
+
+function parseArgs(argv) {
+  const options = {
+    dataset: null,
+    scenarios: null,
+    dimensions: null,
+    questionIds: null,
+    limit: 20,
+    candidateLimit: 100,
+    maxQuestions: null,
+    sourceTags: false,
+    embeddingModule: null,
+    json: false,
+    out: null,
+    strict: false,
+    quiet: false,
+  };
+  const list = (value) =>
+    String(value ?? '')
+      .split(',')
+      .map((item) => item.trim())
+      .filter(Boolean);
+
+  for (let index = 0; index < argv.length; index += 1) {
+    const arg = argv[index];
+    const next = () => argv[(index += 1)];
+    switch (arg) {
+      case '--dataset': options.dataset = next(); break;
+      case '--scenario': options.scenarios = list(next()); break;
+      case '--dimension': options.dimensions = list(next()); break;
+      case '--question': options.questionIds = list(next()); break;
+      case '--limit': options.limit = Number(next()); break;
+      case '--candidate-limit': options.candidateLimit = Number(next()); break;
+      case '--max-questions': options.maxQuestions = Number(next()); break;
+      case '--source-tags': options.sourceTags = true; break;
+      case '--embedding-module': options.embeddingModule = next(); break;
+      case '--json': options.json = true; break;
+      case '--out': options.out = next(); break;
+      case '--strict': options.strict = true; break;
+      case '--quiet': options.quiet = true; break;
+      case '-h':
+      case '--help': options.help = true; break;
+      default:
+        if (arg.startsWith('--')) throw new Error(`Unknown flag: ${arg}`);
+    }
+  }
+  return options;
+}
+
+function usage() {
+  process.stdout.write(`Usage: node scripts/run-memory-v2-memfuse-bench.mjs [options]
+
+  --dataset <path>          MemFuseBench dataset JSON (default: $MEMFUSE_BENCH_DATASET)
+  --scenario <ids>          Comma-separated scenario ids, e.g. sc1,sc2
+  --dimension <names>       Comma-separated dimensions. Known values:
+                            ${MEMFUSE_DIMENSIONS.join('\n                            ')}
+  --question <ids>          Comma-separated question ids
+  --limit <n>               Retrieved rows per question (clamped to 50)
+  --candidate-limit <n>     Candidate pool size before ranking (clamped to 100)
+  --max-questions <n>       Cap questions per scenario (useful for smoke runs)
+  --source-tags             Prepend "[device · location]" to event text
+  --embedding-module <spec> Module exporting embedQuery/embedText/default
+  --json                    Print the full report as JSON
+  --out <path>              Write the JSON report to a file
+  --strict                  Exit non-zero when the dataset is unavailable
+  --quiet                   Suppress per-scenario progress
+`);
+}
+
+async function loadEmbedder(specifier) {
+  const resolved = resolveEmbeddingModuleSpecifier(specifier);
+  if (!resolved) throw new Error(`Cannot resolve embedding module: ${specifier}`);
+  const imported = await import(resolved);
+  const embed = imported?.embedText ?? imported?.embedQuery ?? imported?.default;
+  if (typeof embed !== 'function') {
+    throw new Error(`Embedding module must export embedText, embedQuery or default: ${specifier}`);
+  }
+  return embed;
+}
+
+function formatPercent(value) {
+  return Number.isFinite(value) ? `${(value * 100).toFixed(1)}%` : 'n/a';
+}
+
+function formatRow(label, metrics, width) {
+  return [
+    label.padEnd(width),
+    String(metrics.caseCount).padStart(5),
+    formatPercent(metrics.candidateRecall).padStart(9),
+    formatPercent(metrics.recallAtK).padStart(8),
+    formatPercent(metrics.rankingLoss).padStart(8),
+    formatPercent(metrics.hitAnyRate).padStart(8),
+    formatPercent(metrics.checklistCoverage).padStart(10),
+    (Number.isFinite(metrics.mrr) ? metrics.mrr.toFixed(3) : 'n/a').padStart(7),
+    formatPercent(metrics.distractorRate).padStart(11),
+  ].join('  ');
+}
+
+function printReport(report) {
+  const labels = [
+    ...report.byScenario.map((item) => item.scenarioId),
+    ...Object.keys(report.byDimension),
+    'OVERALL',
+  ];
+  const width = Math.max(12, ...labels.map((label) => label.length));
+  const header = [
+    'group'.padEnd(width),
+    'cases'.padStart(5),
+    'cand.rec'.padStart(9),
+    'recall'.padStart(8),
+    'rankLoss'.padStart(8),
+    'hit@k'.padStart(8),
+    'checklist'.padStart(10),
+    'mrr'.padStart(7),
+    'distractor'.padStart(11),
+  ].join('  ');
+
+  process.stdout.write(`\nMemFuseBench retrieval @k=${report.limit}`);
+  process.stdout.write(
+    ` · embedding=${report.embeddingMode}` +
+      ` · sourceTags=${report.includeSourceTags ? 'on' : 'off'}\n`,
+  );
+  if (report.embeddingMode === 'lexical-hash') {
+    process.stdout.write(
+      'NOTE: lexical-hash is a deterministic offline stand-in, not a semantic embedder.\n' +
+        '      Use --embedding-module to measure real semantic recall.\n',
+    );
+  }
+  process.stdout.write(`\n${header}\n${'-'.repeat(header.length)}\n`);
+  for (const scenario of report.byScenario) {
+    process.stdout.write(`${formatRow(scenario.scenarioId, scenario, width)}\n`);
+  }
+  process.stdout.write(`${'-'.repeat(header.length)}\n`);
+  for (const [dimension, metrics] of Object.entries(report.byDimension).sort(
+    (left, right) => left[1].recallAtK - right[1].recallAtK,
+  )) {
+    process.stdout.write(`${formatRow(dimension, metrics, width)}\n`);
+  }
+  process.stdout.write(`${'-'.repeat(header.length)}\n`);
+  process.stdout.write(`${formatRow('OVERALL', report.overall, width)}\n\n`);
+}
+
+async function main() {
+  const options = parseArgs(process.argv.slice(2));
+  if (options.help) {
+    usage();
+    return 0;
+  }
+
+  const loaded = await loadMemFuseDataset({ datasetPath: options.dataset });
+  if (!loaded.available) {
+    const message =
+      `MemFuseBench dataset unavailable (${loaded.reason}) at ${loaded.path}\n` +
+      'Clone it outside this repo, then set MEMFUSE_BENCH_DATASET or pass --dataset:\n' +
+      '  git clone https://github.com/Darwin-Agent/Mi-Memory ~/Project/mi-memory\n';
+    process.stderr.write(message);
+    return options.strict ? 1 : 0;
+  }
+
+  if (!options.quiet) {
+    process.stdout.write(
+      `dataset: ${loaded.path}\n` +
+        `scenarios=${loaded.stats.scenarioCount} ` +
+        `events=${loaded.stats.eventCount} questions=${loaded.stats.questionCount}\n`,
+    );
+  }
+
+  const embedText = options.embeddingModule
+    ? await loadEmbedder(options.embeddingModule)
+    : null;
+
+  let lastScenario = null;
+  const report = await runMemFuseBench({
+    dataset: loaded,
+    scenarioIds: options.scenarios,
+    dimensions: options.dimensions,
+    questionIds: options.questionIds,
+    maxQuestionsPerScenario: Number.isFinite(options.maxQuestions) ? options.maxQuestions : null,
+    limit: options.limit,
+    candidateLimit: options.candidateLimit,
+    includeSourceTags: options.sourceTags,
+    embedText,
+    onProgress: options.quiet
+      ? null
+      : (event) => {
+          if (event.scenarioId !== lastScenario) {
+            lastScenario = event.scenarioId;
+            process.stderr.write(`  running ${event.scenarioId}...\n`);
+          }
+        },
+  });
+
+  if (options.json) {
+    process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
+  } else {
+    printReport(report);
+    process.stdout.write(`${JSON.stringify(summarizeMemFuseBench(report), null, 2)}\n`);
+  }
+
+  if (options.out) {
+    const target = path.resolve(options.out);
+    await fs.mkdir(path.dirname(target), { recursive: true });
+    await fs.writeFile(target, `${JSON.stringify(report, null, 2)}\n`, 'utf8');
+    process.stderr.write(`report written to ${target}\n`);
+  }
+
+  return 0;
+}
+
+main()
+  .then((code) => process.exit(code))
+  .catch((err) => {
+    process.stderr.write(`${err instanceof Error ? err.stack : String(err)}\n`);
+    process.exit(1);
+  });