Add MemFuseBench retrieval harness and first baseline report.
Wire an offline benchmark over the production pgvector ranking path so Memory V2 recall can be measured with candidate/ranking loss split before lifecycle work lands. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
john
@@ -0,0 +1,132 @@
|
||||
# MemFuseBench 检索基线（Memory V2）
|
||||
|
||||
日期: 2026-09-02
|
||||
|
||||
状态: 已落地测量工具 + 首份基线。**不改动任何生产运行时路径**，纯离线评测。
|
||||
|
||||
## 0. 结论
|
||||
|
||||
在 MemFuseBench 的 357 个证据锚定问题、7823 条来源标注事件上跑现网 pgvector 检索排序链路，得到：
|
||||
|
||||
| 指标 | k=5 | k=20 | k=50 |
|
||||
|---|---|---|---|
|
||||
| 候选召回 candidateRecall | 36.6% | 36.6% | 36.6% |
|
||||
| 最终召回 recall@k | 6.8% | 13.5% | 22.5% |
|
||||
| **排序丢失 rankingLoss** | **29.8%** | **23.1%** | **14.2%** |
|
||||
| 至少命中一条 hitAny@k | 21.8% | 41.7% | 64.4% |
|
||||
| 清单覆盖 checklistCoverage | 7.8% | 16.8% | 28.3% |
|
||||
| MRR | 0.135 | 0.153 | 0.160 |
|
||||
| 干扰项占比 distractorRate | 8.5% | 10.8% | 12.6% |
|
||||
|
||||
两个可操作结论：
|
||||
|
||||
1. **天花板在候选生成，不在 k。** candidateRecall 恒为 36.6%，与 k 无关——超过 63% 的证据从未进入候选池，再怎么调排序或放大 k 都拿不回来。
|
||||
2. **进了候选池的证据，排序又丢掉了一大半。** k=20 时候选池已包含 36.6% 的证据，最终只留下 13.5%，rankingLoss 23.1 个百分点。即使 k=50（在约 200 条候选里返回 50 条），仍丢 14.2 个百分点。MRR 从 k=5 到 k=50 只从 0.135 涨到 0.160，说明列表头部的质量与 k 无关地差。
|
||||
|
||||
按维度看，`multi_source_conflict_arbitration` 明显最好（候选 71.5% / 召回 39.4%），`cross_device_information_fusion` 最差（26.5% / 6.1%）。
|
||||
|
||||
## 1. 两个已定位的机制
|
||||
|
||||
### 机制一：`lexicalQueryCoverage` 是主排序键，但判别力接近随机
|
||||
|
||||
`memory-v2-pgvector.mjs` 的 `rankHybridCandidates` 把 `lexicalScore` 作为**第一排序键**，向量分只在 lexical 分相同时才起作用。而 `lexicalQueryCoverage` 是字符 2-gram 覆盖率。全量 357 题实测：
|
||||
|
||||
- gold 事件平均 coverage：`0.5366`
|
||||
- 非 gold 平均 coverage：`0.4853`
|
||||
- gold 高于语料中位数的比例：`64.5%`（随机为 50%）
|
||||
|
||||
也就是一个只比随机好 14.5 个百分点的弱信号，被用作压倒性的主排序键。在英文语料上排名最高的往往是最长、bigram 最密的无关句子。
|
||||
|
||||
这个启发式对中文是合理的——CJK 2-gram 近似于词，判别力强；对英文退化严重，因为 `th`/`he`/`in` 这类 bigram 在任何长句里都存在。
|
||||
|
||||
### 机制二：keyword 回退路径按时间排序截断，不按相关性
|
||||
|
||||
`fetchKeywordCandidates` 的 SQL 是 `... WHERE content ILIKE ANY(...) ORDER BY updated_at DESC LIMIT n`。全量实测：
|
||||
|
||||
- ILIKE 能匹配到的 gold：`60.4%`
|
||||
- 按 `updated_at DESC` 截断 100 条后存活的 gold：`14.2%`
|
||||
- 至少存活一条 gold 的问题占比：`32.5%`
|
||||
|
||||
一次纯粹由"按时间截断"造成的 46 个百分点损失。语料里匹配关键词的事件一旦多于 limit，这条路召回的就是"最新的匹配"而不是"最相关的匹配"。
|
||||
|
||||
## 2. 必须一起读的三条限制
|
||||
|
||||
不要把上面的绝对数字当成生产召回率。
|
||||
|
||||
1. **默认嵌入是 `lexical-hash`，不是语义嵌入。** 它是哈希词袋 + L2 归一化的确定性离线替身，cosine 近似词面重叠。报告里 `embeddingMode` 会标出来。机制一和机制二与嵌入无关（都在 lexical / keyword 路径上），但 36.6% 这个候选天花板里有一部分要归因于替身嵌入弱。用 `--embedding-module` 接真实嵌入后才能给向量路径定论。
|
||||
2. **语料是英文，而 lexical 层是按中文调的。** 见机制一。这既是限制也是发现：一旦 Memind 要处理非中文内容，这条排序键就失效。
|
||||
3. **语料粒度比现网粗。** MemFuseBench 的语料是事件级（单场景 936–1886 条原始事件），而现网 `memory_embeddings` 存的是已经筛过的用户记忆，条数少得多。所以这份基线衡量的是"如果把原始事件流直接灌进现有检索层会怎样"——正好是 Event Kernel 方向会产生的形态。
|
||||
|
||||
## 3. 数据集与授权
|
||||
|
||||
**数据集不入库。** 它在仓库外，通过 `MEMFUSE_BENCH_DATASET` 或 `--dataset` 指定；缺失时所有入口返回 `{ available: false, reason }`，CLI 打印 skip 并 exit 0（`--strict` 才 exit 1）。CI 无需数据集即可全绿。
|
||||
|
||||
```bash
|
||||
git clone https://github.com/Darwin-Agent/Mi-Memory ~/Project/mi-memory
|
||||
```
|
||||
|
||||
默认路径 `~/Project/mi-memory/MemFuse/MemFuseBench/memfusebench_dataset.json`（6.2MB）。
|
||||
|
||||
| 项目 | 授权 | 本仓库可以做什么 |
|
||||
|---|---|---|
|
||||
| Mi-Memory / MemFuseBench | MIT（Darwin Agent Team, Xiaomi） | 可自由使用数据集与论文思想 |
|
||||
| JKRiver / Riverse | **AGPL-3.0 或商业双授权** | **只做架构级参考，禁止移植代码** |
|
||||
|
||||
JKRiver 的 `LICENSE` 明确把"集成进专有产品"和"作为商业 SaaS 提供而不开源修改"列为需要单独商业授权的场景。Memind 是带计费的商业 SaaS，AGPL 的网络 copyleft 正好命中。因此：
|
||||
|
||||
- 允许：借鉴设计决策（如整条流水线单事务 + 末步落 processed 水位的 at-least-once 幂等）、参考反面经验（`migrations/008_drop_hypotheses.sql` 记录了 hypotheses 表建后即废，被 `layer='suspected'/'confirmed'` + supersedes 取代）。
|
||||
- 禁止：把 `agent/sleep/*.py`（含 `_maturity.py` 的分层衰减表、`disputes.py` 的争议解决）逐行翻译进 `memory-v2-lifecycle.mjs`。取得商业授权前不做。
|
||||
|
||||
Mi-Memory 仓库本身**没有代码**，只有论文 PDF、项目页和图；MemStack / MemSense / D²ACCI / E²MEND / LiteMem 均无参考实现。唯一可直接使用的产物就是 MemFuseBench 数据集。
|
||||
|
||||
## 4. 用法
|
||||
|
||||
```bash
|
||||
# 全量 357 题，约 3.5s
|
||||
npm run bench:memory-v2-memfuse
|
||||
|
||||
# 单场景 / 单维度 / 冒烟
|
||||
node scripts/run-memory-v2-memfuse-bench.mjs --scenario sc1 --limit 20
|
||||
node scripts/run-memory-v2-memfuse-bench.mjs --dimension multi_source_conflict_arbitration
|
||||
node scripts/run-memory-v2-memfuse-bench.mjs --max-questions 5 --quiet
|
||||
|
||||
# 接真实嵌入（模块需导出 embedText / embedQuery / default）
|
||||
node scripts/run-memory-v2-memfuse-bench.mjs --embedding-module ./scripts/my-embedder.mjs
|
||||
|
||||
# 事件文本前置 "[device · location]" 来源标签
|
||||
node scripts/run-memory-v2-memfuse-bench.mjs --source-tags
|
||||
|
||||
# 落 JSON 报告
|
||||
node scripts/run-memory-v2-memfuse-bench.mjs --out .release-gate/memfuse.json
|
||||
```
|
||||
|
||||
`--source-tags` 实测把候选召回从 36.6% 抬到 38.2%，但最终召回不变（13.5%），说明当前排序器吃不到 provenance 前缀带来的信息。
|
||||
|
||||
单测（不依赖外部数据集）：
|
||||
|
||||
```bash
|
||||
npm run verify:memory-v2-memfuse-bench
|
||||
```
|
||||
|
||||
## 5. 指标定义
|
||||
|
||||
- `candidateRecall` — 证据事件进入候选池（向量 top-N ∪ 最近 N ∪ keyword 匹配）的比例。**这是排序无关的天花板。**
|
||||
- `recall@k` — 证据事件出现在最终 top-k 的比例。
|
||||
- `rankingLoss` = `candidateRecall - recall@k`。大于 0 表示证据已被检索到但被排序丢弃。**这是决定"该修检索还是该修排序"的关键拆分**，对应 Mi-Memory 讲的 diagnostic trace。
|
||||
- `hitAny@k` — 至少召回一条证据的问题占比。
|
||||
- `checklistCoverage` — `answer_checklist` 中每个 point 至少召回一条其 `source_events` 的比例。这是最接近 MemFuse 论文自身评分（answer-checklist coverage）的检索侧代理，无需 LLM 评审。
|
||||
- `distractorRate` — top-k 中 `source` 为 `noise` / `adversarial` 的占比。这些事件按数据集构造永远不属于任何答案的证据集。
|
||||
|
||||
## 6. 这份基线解锁了什么
|
||||
|
||||
`memory-v2-lifecycle.mjs` 的 `promote` / `compact` / `reflect` / decay 目前 flag 全关、`reflect()` 恒返回 `updated: 0`。在没有基线之前，往里面填任何算法都无法验证对错。现在有了一个 3.5 秒可重跑、357 题、能把候选损失和排序损失分开的度量，这个前置条件解除。
|
||||
|
||||
按 rankingLoss 的量级，优先级排序是：先修排序（`rankHybridCandidates` 的主键选择、`fetchKeywordCandidates` 的排序方式），再谈候选生成和 consolidation 算法。这两处都是几十行的局部改动，而且每一步都能用本文的表格直接对比。
|
||||
|
||||
## 7. 相关文件
|
||||
|
||||
- `memory-v2-memfuse-bench.mjs` — 数据集加载、语料/案例构建、评测与聚合
|
||||
- `memory-v2-memfuse-bench.test.mjs` — 内联 fixture，19 个用例，不依赖外部数据集
|
||||
- `scripts/run-memory-v2-memfuse-bench.mjs` — CLI
|
||||
- `memory-v2-pgvector.mjs` — 被测的生产排序路径（`rankHybridCandidates` / `fetchKeywordCandidates` / `extractKeywordTerms`）
|
||||
- `memory-v2-recall-benchmark.mjs` — 既有的 4 例中文手写基线，与本文互补
|
||||
@@ -0,0 +1,600 @@
|
||||
import os from 'node:os';
|
||||
import path from 'node:path';
|
||||
import { readFile as fsReadFile } from 'node:fs/promises';
|
||||
|
||||
import { createPgvectorMemoryBackend } from './memory-v2-pgvector.mjs';
|
||||
|
||||
/**
|
||||
* MemFuseBench retrieval harness for Memory V2.
|
||||
*
|
||||
* MemFuseBench (Mi-Memory / Darwin Agent Team, MIT) ships 357 evidence-grounded
|
||||
* questions over 7,823 source-tagged events. This module turns those scenarios
|
||||
* into a corpus + case list and runs them through the *production* pgvector
|
||||
* backend so the measured numbers reflect `rankHybridCandidates` and
|
||||
* `extractKeywordTerms`, not a parallel scoring implementation.
|
||||
*
|
||||
* The dataset itself is never vendored into this repository. It is read from an
|
||||
* external checkout and every entry point degrades to
|
||||
* `{ available: false, reason }` when the file is absent, so CI stays green
|
||||
* without it.
|
||||
*/
|
||||
|
||||
export const MEMFUSE_DATASET_ENV = 'MEMFUSE_BENCH_DATASET';
|
||||
|
||||
export const MEMFUSE_DIMENSIONS = Object.freeze([
|
||||
'cross_device_causal_reasoning',
|
||||
'cross_device_information_fusion',
|
||||
'multi_source_conflict_arbitration',
|
||||
'cross_user_information_synthesis',
|
||||
'cross_user_query',
|
||||
'perspective_difference',
|
||||
]);
|
||||
|
||||
// Events tagged with these sources are adversarial by construction: they look
|
||||
// topical but are never part of any answer's evidence set.
|
||||
export const MEMFUSE_DISTRACTOR_SOURCES = Object.freeze(['noise', 'adversarial']);
|
||||
|
||||
const DEFAULT_RELATIVE_DATASET_PATH = path.join(
|
||||
'mi-memory',
|
||||
'MemFuse',
|
||||
'MemFuseBench',
|
||||
'memfusebench_dataset.json',
|
||||
);
|
||||
|
||||
export function resolveMemFuseDatasetPath(env = process.env) {
|
||||
const explicit = String(env?.[MEMFUSE_DATASET_ENV] ?? '').trim();
|
||||
if (explicit) return path.resolve(explicit);
|
||||
const projectRoot = String(env?.MEMFUSE_PROJECT_ROOT ?? '').trim();
|
||||
const base = projectRoot || path.join(os.homedir(), 'Project');
|
||||
return path.join(base, DEFAULT_RELATIVE_DATASET_PATH);
|
||||
}
|
||||
|
||||
function asArray(value) {
|
||||
return Array.isArray(value) ? value : [];
|
||||
}
|
||||
|
||||
function nonEmptyString(value) {
|
||||
const text = String(value ?? '').trim();
|
||||
return text.length > 0 ? text : null;
|
||||
}
|
||||
|
||||
export function normalizeMemFuseDataset(raw) {
|
||||
const scenarios = asArray(raw?.scenarios)
|
||||
.map((scenario) => {
|
||||
const scenarioId = nonEmptyString(scenario?.scenario_id);
|
||||
if (!scenarioId) return null;
|
||||
const episodes = asArray(scenario?.episodes)
|
||||
.map((episode) => {
|
||||
const episodeId = nonEmptyString(episode?.episode_id);
|
||||
const events = asArray(episode?.events)
|
||||
.map((event) => {
|
||||
const eventId = nonEmptyString(event?.event_id);
|
||||
const description = nonEmptyString(event?.description);
|
||||
if (!eventId || !description) return null;
|
||||
return {
|
||||
eventId,
|
||||
description,
|
||||
device: nonEmptyString(event?.device),
|
||||
modality: nonEmptyString(event?.modality) ?? 'event',
|
||||
location: nonEmptyString(event?.location),
|
||||
source: nonEmptyString(event?.source) ?? 'unknown',
|
||||
timestamp: nonEmptyString(event?.timestamp),
|
||||
characters: asArray(event?.characters).map(String),
|
||||
};
|
||||
})
|
||||
.filter(Boolean);
|
||||
if (!episodeId || events.length === 0) return null;
|
||||
return { episodeId, events };
|
||||
})
|
||||
.filter(Boolean);
|
||||
const questions = asArray(scenario?.questions)
|
||||
.map((question) => {
|
||||
const questionId = nonEmptyString(question?.question_id);
|
||||
const text = nonEmptyString(question?.question);
|
||||
const evidenceEventIds = [
|
||||
...new Set(asArray(question?.evidence_event_ids).map(String).filter(Boolean)),
|
||||
];
|
||||
if (!questionId || !text || evidenceEventIds.length === 0) return null;
|
||||
const checklist = asArray(question?.answer_checklist)
|
||||
.map((entry) => ({
|
||||
point: nonEmptyString(entry?.point) ?? '',
|
||||
sourceEvents: [
|
||||
...new Set(asArray(entry?.source_events).map(String).filter(Boolean)),
|
||||
],
|
||||
}))
|
||||
.filter((entry) => entry.sourceEvents.length > 0);
|
||||
return {
|
||||
questionId,
|
||||
scenarioId,
|
||||
question: text,
|
||||
evidenceEventIds,
|
||||
checklist,
|
||||
dimension: nonEmptyString(question?.dimension) ?? 'unknown',
|
||||
questionUser: nonEmptyString(question?.question_user),
|
||||
questionTime: nonEmptyString(question?.question_time),
|
||||
questionDevice: nonEmptyString(question?.question_device),
|
||||
};
|
||||
})
|
||||
.filter(Boolean);
|
||||
if (episodes.length === 0 || questions.length === 0) return null;
|
||||
return {
|
||||
scenarioId,
|
||||
description: nonEmptyString(scenario?.description),
|
||||
timeSpan: nonEmptyString(scenario?.time_span),
|
||||
episodes,
|
||||
questions,
|
||||
};
|
||||
})
|
||||
.filter(Boolean);
|
||||
|
||||
if (scenarios.length === 0) {
|
||||
return { ok: false, reason: 'dataset_has_no_usable_scenarios' };
|
||||
}
|
||||
|
||||
const eventCount = scenarios.reduce(
|
||||
(total, scenario) =>
|
||||
total + scenario.episodes.reduce((sum, episode) => sum + episode.events.length, 0),
|
||||
0,
|
||||
);
|
||||
const questionCount = scenarios.reduce(
|
||||
(total, scenario) => total + scenario.questions.length,
|
||||
0,
|
||||
);
|
||||
|
||||
return {
|
||||
ok: true,
|
||||
dataset: {
|
||||
scenarios,
|
||||
stats: { scenarioCount: scenarios.length, eventCount, questionCount },
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
export async function loadMemFuseDataset({
|
||||
datasetPath = null,
|
||||
env = process.env,
|
||||
readFile = fsReadFile,
|
||||
} = {}) {
|
||||
const resolvedPath = datasetPath ? path.resolve(datasetPath) : resolveMemFuseDatasetPath(env);
|
||||
let payload;
|
||||
try {
|
||||
payload = await readFile(resolvedPath, 'utf8');
|
||||
} catch (err) {
|
||||
return {
|
||||
available: false,
|
||||
path: resolvedPath,
|
||||
reason: err?.code === 'ENOENT' ? 'dataset_not_found' : 'dataset_read_failed',
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
};
|
||||
}
|
||||
let raw;
|
||||
try {
|
||||
raw = JSON.parse(payload);
|
||||
} catch (err) {
|
||||
return {
|
||||
available: false,
|
||||
path: resolvedPath,
|
||||
reason: 'dataset_parse_failed',
|
||||
error: err instanceof Error ? err.message : String(err),
|
||||
};
|
||||
}
|
||||
const normalized = normalizeMemFuseDataset(raw);
|
||||
if (!normalized.ok) {
|
||||
return { available: false, path: resolvedPath, reason: normalized.reason };
|
||||
}
|
||||
return { available: true, path: resolvedPath, ...normalized.dataset };
|
||||
}
|
||||
|
||||
/**
|
||||
* Flatten a scenario into rows shaped like the `memory_embeddings` table so the
|
||||
* pgvector backend can consume them unchanged.
|
||||
*
|
||||
* `includeSourceTags` prepends `[device · location]`. It defaults to false so the
|
||||
* baseline measures retrieval over raw event text rather than over a prefix
|
||||
* format chosen here.
|
||||
*/
|
||||
export function buildScenarioCorpus(scenario, { includeSourceTags = false } = {}) {
|
||||
const rows = [];
|
||||
const byId = new Map();
|
||||
for (const episode of asArray(scenario?.episodes)) {
|
||||
for (const event of asArray(episode?.events)) {
|
||||
const tags = [event.device, event.location].filter(Boolean).join(' · ');
|
||||
const content =
|
||||
includeSourceTags && tags ? `[${tags}] ${event.description}` : event.description;
|
||||
const timestampMs = event.timestamp ? Date.parse(event.timestamp) : NaN;
|
||||
const isoTimestamp = Number.isFinite(timestampMs)
|
||||
? new Date(timestampMs).toISOString()
|
||||
: null;
|
||||
const row = {
|
||||
id: event.eventId,
|
||||
content,
|
||||
type: event.modality,
|
||||
created_at: isoTimestamp,
|
||||
updated_at: isoTimestamp,
|
||||
episodeId: episode.episodeId,
|
||||
source: event.source,
|
||||
device: event.device,
|
||||
location: event.location,
|
||||
timestampMs: Number.isFinite(timestampMs) ? timestampMs : 0,
|
||||
};
|
||||
rows.push(row);
|
||||
byId.set(row.id, row);
|
||||
}
|
||||
}
|
||||
return { rows, byId };
|
||||
}
|
||||
|
||||
export function buildScenarioCases(
|
||||
scenario,
|
||||
{ dimensions = null, maxQuestions = null, questionIds = null } = {},
|
||||
) {
|
||||
const dimensionFilter = dimensions?.length ? new Set(dimensions) : null;
|
||||
const idFilter = questionIds?.length ? new Set(questionIds) : null;
|
||||
const cases = asArray(scenario?.questions).filter((question) => {
|
||||
if (dimensionFilter && !dimensionFilter.has(question.dimension)) return false;
|
||||
if (idFilter && !idFilter.has(question.questionId)) return false;
|
||||
return true;
|
||||
});
|
||||
const capped = Number(maxQuestions);
|
||||
return Number.isFinite(capped) && capped > 0 ? cases.slice(0, capped) : cases;
|
||||
}
|
||||
|
||||
const CJK_RANGE = /[\u4e00-\u9fff]/u;
|
||||
|
||||
function tokenizeForEmbedding(text) {
|
||||
const normalized = String(text ?? '')
|
||||
.normalize('NFKC')
|
||||
.toLowerCase();
|
||||
const tokens = [];
|
||||
for (const match of normalized.matchAll(/[a-z0-9]{2,}/g)) tokens.push(match[0]);
|
||||
if (CJK_RANGE.test(normalized)) {
|
||||
const cjk = normalized.replace(/[^\u4e00-\u9fff]/gu, '');
|
||||
for (let index = 0; index + 2 <= cjk.length; index += 1) {
|
||||
tokens.push(cjk.slice(index, index + 2));
|
||||
}
|
||||
}
|
||||
return tokens;
|
||||
}
|
||||
|
||||
function hashToken(token, dimensions) {
|
||||
let hash = 2166136261;
|
||||
for (let index = 0; index < token.length; index += 1) {
|
||||
hash ^= token.charCodeAt(index);
|
||||
hash = Math.imul(hash, 16777619);
|
||||
}
|
||||
return Math.abs(hash) % dimensions;
|
||||
}
|
||||
|
||||
/**
|
||||
* Deterministic offline stand-in for a sentence embedder: hashed bag-of-tokens,
|
||||
* L2 normalized. Cosine over these vectors approximates lexical overlap, NOT
|
||||
* semantic similarity, so reports built on it are labelled `lexical-hash`. Pass
|
||||
* a real `embedText` to measure semantic recall.
|
||||
*/
|
||||
export function createLexicalHashEmbedder({ dimensions = 256 } = {}) {
|
||||
const size = Math.max(16, Math.min(4096, Number(dimensions) || 256));
|
||||
return function embedText(text) {
|
||||
const vector = new Array(size).fill(0);
|
||||
const tokens = tokenizeForEmbedding(text);
|
||||
if (tokens.length === 0) return vector;
|
||||
for (const token of tokens) {
|
||||
vector[hashToken(token, size)] += 1;
|
||||
}
|
||||
let norm = 0;
|
||||
for (const value of vector) norm += value * value;
|
||||
norm = Math.sqrt(norm);
|
||||
if (norm === 0) return vector;
|
||||
for (let index = 0; index < size; index += 1) vector[index] /= norm;
|
||||
return vector;
|
||||
};
|
||||
}
|
||||
|
||||
function cosineSimilarity(left, right) {
|
||||
const length = Math.min(left.length, right.length);
|
||||
let dot = 0;
|
||||
let leftNorm = 0;
|
||||
let rightNorm = 0;
|
||||
for (let index = 0; index < length; index += 1) {
|
||||
dot += left[index] * right[index];
|
||||
leftNorm += left[index] * left[index];
|
||||
rightNorm += right[index] * right[index];
|
||||
}
|
||||
if (leftNorm === 0 || rightNorm === 0) return 0;
|
||||
return dot / (Math.sqrt(leftNorm) * Math.sqrt(rightNorm));
|
||||
}
|
||||
|
||||
function parseVectorLiteral(literal) {
|
||||
const text = String(literal ?? '').trim();
|
||||
if (!text.startsWith('[') || !text.endsWith(']')) return null;
|
||||
const inner = text.slice(1, -1);
|
||||
if (!inner) return [];
|
||||
const parts = inner.split(',').map((part) => Number(part));
|
||||
return parts.some((part) => !Number.isFinite(part)) ? null : parts;
|
||||
}
|
||||
|
||||
function likeParamToTerm(param) {
|
||||
return String(param ?? '')
|
||||
.replace(/^%+/, '')
|
||||
.replace(/%+$/, '')
|
||||
.toLowerCase();
|
||||
}
|
||||
|
||||
/**
|
||||
* Fake pg pool that emulates the two query shapes issued by
|
||||
* `createPgvectorMemoryBackend.resolve`: the vector/recent candidate union and
|
||||
* the `content ILIKE` keyword fallback. Emulating the SQL rather than bypassing
|
||||
* it keeps the production ranking path under test.
|
||||
*/
|
||||
export function createCorpusPool({ rows, embedText, embeddingCache = new Map() }) {
|
||||
const corpus = asArray(rows);
|
||||
const byRecency = [...corpus].sort((left, right) => right.timestampMs - left.timestampMs);
|
||||
|
||||
function embeddingFor(row) {
|
||||
if (!embeddingCache.has(row.id)) {
|
||||
embeddingCache.set(row.id, embedText(row.content));
|
||||
}
|
||||
return embeddingCache.get(row.id);
|
||||
}
|
||||
|
||||
function toResultRow(row, score) {
|
||||
return {
|
||||
id: row.id,
|
||||
content: row.content,
|
||||
type: row.type,
|
||||
created_at: row.created_at,
|
||||
updated_at: row.updated_at,
|
||||
score,
|
||||
};
|
||||
}
|
||||
|
||||
let vectorQueryCount = 0;
|
||||
let keywordQueryCount = 0;
|
||||
// Every id handed back to the backend, i.e. the candidate pool that ranking
|
||||
// then narrows to top-k. Tracking it separates "candidate generation lost the
|
||||
// evidence" from "ranking lost the evidence".
|
||||
const returnedIds = new Set();
|
||||
|
||||
function trackRows(rows) {
|
||||
for (const row of rows) returnedIds.add(row.id);
|
||||
return rows;
|
||||
}
|
||||
|
||||
return {
|
||||
stats: () => ({
|
||||
vectorQueryCount,
|
||||
keywordQueryCount,
|
||||
corpusSize: corpus.length,
|
||||
candidateIds: new Set(returnedIds),
|
||||
}),
|
||||
async query(sql, params = []) {
|
||||
const text = String(sql ?? '');
|
||||
if (text.includes('vector_candidates')) {
|
||||
vectorQueryCount += 1;
|
||||
const queryVector = parseVectorLiteral(params[1]);
|
||||
const limit = Math.max(1, Number(params[2]) || 50);
|
||||
const scored = corpus.map((row) => ({
|
||||
row,
|
||||
score: queryVector ? cosineSimilarity(queryVector, embeddingFor(row)) : 0,
|
||||
}));
|
||||
const scoreById = new Map(scored.map((entry) => [entry.row.id, entry.score]));
|
||||
const vectorTop = [...scored]
|
||||
.sort((left, right) => right.score - left.score)
|
||||
.slice(0, limit)
|
||||
.map((entry) => entry.row);
|
||||
const recentTop = byRecency.slice(0, limit);
|
||||
const merged = new Map();
|
||||
for (const row of [...vectorTop, ...recentTop]) {
|
||||
if (!merged.has(row.id)) merged.set(row.id, row);
|
||||
}
|
||||
return {
|
||||
rows: trackRows(
|
||||
[...merged.values()].map((row) => toResultRow(row, scoreById.get(row.id) ?? 0)),
|
||||
),
|
||||
};
|
||||
}
|
||||
if (text.includes('ILIKE')) {
|
||||
keywordQueryCount += 1;
|
||||
const limit = Math.max(1, Number(params[params.length - 1]) || 20);
|
||||
const terms = params
|
||||
.slice(1, params.length - 1)
|
||||
.map(likeParamToTerm)
|
||||
.filter(Boolean);
|
||||
if (terms.length === 0) return { rows: [] };
|
||||
const matched = byRecency.filter((row) => {
|
||||
const haystack = row.content.toLowerCase();
|
||||
return terms.some((term) => haystack.includes(term));
|
||||
});
|
||||
return { rows: trackRows(matched.slice(0, limit).map((row) => toResultRow(row, 1))) };
|
||||
}
|
||||
return { rows: [] };
|
||||
},
|
||||
};
|
||||
}
|
||||
|
||||
function checklistCoverage(checklist, retrievedIds) {
|
||||
if (!checklist?.length) return null;
|
||||
let covered = 0;
|
||||
for (const entry of checklist) {
|
||||
if (entry.sourceEvents.some((eventId) => retrievedIds.has(eventId))) covered += 1;
|
||||
}
|
||||
return covered / checklist.length;
|
||||
}
|
||||
|
||||
export async function runMemFuseBenchCase({
|
||||
testCase,
|
||||
corpus,
|
||||
embedText,
|
||||
limit = 20,
|
||||
candidateLimit = 100,
|
||||
embeddingCache = new Map(),
|
||||
}) {
|
||||
const pool = createCorpusPool({ rows: corpus.rows, embedText, embeddingCache });
|
||||
const backend = createPgvectorMemoryBackend({ enabled: true, embedQuery: embedText, pool });
|
||||
const result = await backend.resolve({
|
||||
userId: 'memfuse-bench-user',
|
||||
query: testCase.question,
|
||||
limit,
|
||||
candidateLimit,
|
||||
});
|
||||
|
||||
const retrieved = result.memories.map((memory) => memory.id).filter(Boolean);
|
||||
const retrievedIds = new Set(retrieved);
|
||||
const gold = new Set(testCase.evidenceEventIds);
|
||||
const hits = retrieved.filter((id) => gold.has(id));
|
||||
const firstHitIndex = retrieved.findIndex((id) => gold.has(id));
|
||||
const distractors = retrieved.filter((id) =>
|
||||
MEMFUSE_DISTRACTOR_SOURCES.includes(corpus.byId.get(id)?.source),
|
||||
);
|
||||
|
||||
const { candidateIds } = pool.stats();
|
||||
const candidateHits = [...gold].filter((id) => candidateIds.has(id));
|
||||
const candidateRecall = gold.size > 0 ? candidateHits.length / gold.size : 0;
|
||||
const recall = gold.size > 0 ? hits.length / gold.size : 0;
|
||||
|
||||
return {
|
||||
questionId: testCase.questionId,
|
||||
scenarioId: testCase.scenarioId,
|
||||
dimension: testCase.dimension,
|
||||
goldCount: gold.size,
|
||||
returned: retrieved.length,
|
||||
hitCount: hits.length,
|
||||
recall,
|
||||
precision: retrieved.length > 0 ? hits.length / retrieved.length : 0,
|
||||
hitAny: hits.length > 0,
|
||||
candidateCount: candidateIds.size,
|
||||
candidateHitCount: candidateHits.length,
|
||||
candidateRecall,
|
||||
// Positive value = evidence reached the candidate pool but ranking dropped
|
||||
// it. This is the split that decides whether to fix retrieval or ranking.
|
||||
rankingLoss: Math.max(0, candidateRecall - recall),
|
||||
firstHitRank: firstHitIndex >= 0 ? firstHitIndex + 1 : null,
|
||||
reciprocalRank: firstHitIndex >= 0 ? 1 / (firstHitIndex + 1) : 0,
|
||||
checklistCoverage: checklistCoverage(testCase.checklist, retrievedIds),
|
||||
distractorCount: distractors.length,
|
||||
};
|
||||
}
|
||||
|
||||
function mean(values) {
|
||||
const usable = values.filter((value) => Number.isFinite(value));
|
||||
if (usable.length === 0) return 0;
|
||||
return usable.reduce((total, value) => total + value, 0) / usable.length;
|
||||
}
|
||||
|
||||
function aggregate(results) {
|
||||
const coverage = results
|
||||
.map((item) => item.checklistCoverage)
|
||||
.filter((value) => Number.isFinite(value));
|
||||
return {
|
||||
caseCount: results.length,
|
||||
recallAtK: mean(results.map((item) => item.recall)),
|
||||
precisionAtK: mean(results.map((item) => item.precision)),
|
||||
candidateRecall: mean(results.map((item) => item.candidateRecall)),
|
||||
rankingLoss: mean(results.map((item) => item.rankingLoss)),
|
||||
hitAnyRate: results.length
|
||||
? results.filter((item) => item.hitAny).length / results.length
|
||||
: 0,
|
||||
checklistCoverage: coverage.length ? mean(coverage) : null,
|
||||
mrr: mean(results.map((item) => item.reciprocalRank)),
|
||||
distractorRate: results.length
|
||||
? mean(results.map((item) => (item.returned ? item.distractorCount / item.returned : 0)))
|
||||
: 0,
|
||||
};
|
||||
}
|
||||
|
||||
export async function runMemFuseBench({
|
||||
dataset,
|
||||
scenarioIds = null,
|
||||
dimensions = null,
|
||||
maxQuestionsPerScenario = null,
|
||||
questionIds = null,
|
||||
limit = 20,
|
||||
candidateLimit = 100,
|
||||
embedText = null,
|
||||
includeSourceTags = false,
|
||||
onProgress = null,
|
||||
} = {}) {
|
||||
const scenarios = asArray(dataset?.scenarios);
|
||||
if (scenarios.length === 0) throw new Error('runMemFuseBench requires a loaded dataset');
|
||||
const embedder = typeof embedText === 'function' ? embedText : createLexicalHashEmbedder();
|
||||
const embeddingMode = typeof embedText === 'function' ? 'external' : 'lexical-hash';
|
||||
const scenarioFilter = scenarioIds?.length ? new Set(scenarioIds) : null;
|
||||
// The pgvector backend clamps its own limit to 50; mirror it so the reported
|
||||
// `limit` never overstates how many rows were actually considered.
|
||||
const effectiveLimit = Math.max(1, Math.min(50, Number(limit) || 20));
|
||||
|
||||
const results = [];
|
||||
const scenarioReports = [];
|
||||
for (const scenario of scenarios) {
|
||||
if (scenarioFilter && !scenarioFilter.has(scenario.scenarioId)) continue;
|
||||
const cases = buildScenarioCases(scenario, {
|
||||
dimensions,
|
||||
maxQuestions: maxQuestionsPerScenario,
|
||||
questionIds,
|
||||
});
|
||||
if (cases.length === 0) continue;
|
||||
const corpus = buildScenarioCorpus(scenario, { includeSourceTags });
|
||||
// One cache per scenario: corpus embeddings are reused across that
|
||||
// scenario's questions, which is where nearly all the cost sits.
|
||||
const embeddingCache = new Map();
|
||||
const scenarioResults = [];
|
||||
for (const testCase of cases) {
|
||||
const caseResult = await runMemFuseBenchCase({
|
||||
testCase,
|
||||
corpus,
|
||||
embedText: embedder,
|
||||
limit: effectiveLimit,
|
||||
candidateLimit,
|
||||
embeddingCache,
|
||||
});
|
||||
scenarioResults.push(caseResult);
|
||||
results.push(caseResult);
|
||||
if (typeof onProgress === 'function') {
|
||||
onProgress({ scenarioId: scenario.scenarioId, completed: results.length, ...caseResult });
|
||||
}
|
||||
}
|
||||
scenarioReports.push({
|
||||
scenarioId: scenario.scenarioId,
|
||||
corpusSize: corpus.rows.length,
|
||||
...aggregate(scenarioResults),
|
||||
});
|
||||
}
|
||||
|
||||
if (results.length === 0) throw new Error('runMemFuseBench selected zero questions');
|
||||
|
||||
const byDimension = {};
|
||||
for (const dimension of new Set(results.map((item) => item.dimension))) {
|
||||
byDimension[dimension] = aggregate(results.filter((item) => item.dimension === dimension));
|
||||
}
|
||||
|
||||
return {
|
||||
limit: effectiveLimit,
|
||||
candidateLimit,
|
||||
embeddingMode,
|
||||
includeSourceTags,
|
||||
overall: aggregate(results),
|
||||
byScenario: scenarioReports,
|
||||
byDimension,
|
||||
results,
|
||||
};
|
||||
}
|
||||
|
||||
export function summarizeMemFuseBench(report) {
|
||||
const round = (value) => (Number.isFinite(value) ? Number(value.toFixed(4)) : null);
|
||||
return {
|
||||
limit: report.limit,
|
||||
embeddingMode: report.embeddingMode,
|
||||
caseCount: report.overall.caseCount,
|
||||
recallAtK: round(report.overall.recallAtK),
|
||||
precisionAtK: round(report.overall.precisionAtK),
|
||||
candidateRecall: round(report.overall.candidateRecall),
|
||||
rankingLoss: round(report.overall.rankingLoss),
|
||||
hitAnyRate: round(report.overall.hitAnyRate),
|
||||
checklistCoverage: round(report.overall.checklistCoverage),
|
||||
mrr: round(report.overall.mrr),
|
||||
distractorRate: round(report.overall.distractorRate),
|
||||
weakestDimension:
|
||||
Object.entries(report.byDimension)
|
||||
.sort((left, right) => left[1].recallAtK - right[1].recallAtK)
|
||||
.map(([dimension]) => dimension)[0] ?? null,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,449 @@
|
||||
import assert from 'node:assert/strict';
|
||||
import test from 'node:test';
|
||||
|
||||
import {
|
||||
MEMFUSE_DATASET_ENV,
|
||||
buildScenarioCases,
|
||||
buildScenarioCorpus,
|
||||
createCorpusPool,
|
||||
createLexicalHashEmbedder,
|
||||
loadMemFuseDataset,
|
||||
normalizeMemFuseDataset,
|
||||
resolveMemFuseDatasetPath,
|
||||
runMemFuseBench,
|
||||
runMemFuseBenchCase,
|
||||
summarizeMemFuseBench,
|
||||
} from './memory-v2-memfuse-bench.mjs';
|
||||
|
||||
// Mirrors the real MemFuseBench shape (scenarios → episodes → events, plus
|
||||
// questions carrying evidence_event_ids and answer_checklist) at a size that
|
||||
// keeps this suite hermetic: the 6MB upstream dataset is never required.
|
||||
function makeRawDataset() {
|
||||
return {
|
||||
metadata: { total_questions: 3 },
|
||||
scenarios: [
|
||||
{
|
||||
scenario_id: 'sc1',
|
||||
description: 'Scenario sc1',
|
||||
time_span: '2026-05-11 ~ 2026-05-12',
|
||||
episodes: [
|
||||
{
|
||||
episode_id: 'sc1_ep1',
|
||||
events: [
|
||||
{
|
||||
event_id: 'sc1_ep1_e1',
|
||||
device: 'thermostat_living',
|
||||
modality: 'environment',
|
||||
characters: [],
|
||||
description: 'Living room thermostat reading: temperature 21.8C, humidity 47%.',
|
||||
timestamp: '2026-05-11T06:00:00',
|
||||
location: 'living_room',
|
||||
source: 'periodic',
|
||||
},
|
||||
{
|
||||
event_id: 'sc1_ep1_e2',
|
||||
device: 'purifier_living',
|
||||
modality: 'appliance',
|
||||
characters: ['Sarah'],
|
||||
description: 'Sarah switched the air purifier to turbo because of pollen.',
|
||||
timestamp: '2026-05-11T08:12:00',
|
||||
location: 'living_room',
|
||||
source: 'storyline',
|
||||
},
|
||||
{
|
||||
event_id: 'sc1_ep1_e3',
|
||||
device: 'curtain_living',
|
||||
modality: 'appliance',
|
||||
characters: ['Sarah'],
|
||||
description: 'Smart curtains closed to reduce pollen entering the living room.',
|
||||
timestamp: '2026-05-11T08:15:00',
|
||||
location: 'living_room',
|
||||
source: 'storyline',
|
||||
},
|
||||
{
|
||||
event_id: 'sc1_ep1_e4',
|
||||
device: 'phone_david',
|
||||
modality: 'message',
|
||||
characters: ['David'],
|
||||
description: 'Unrelated chatter about a weekend hiking trip to the coast.',
|
||||
timestamp: '2026-05-11T09:00:00',
|
||||
location: 'office',
|
||||
source: 'noise',
|
||||
},
|
||||
{
|
||||
event_id: 'sc1_ep1_e5',
|
||||
device: 'speaker_kitchen',
|
||||
modality: 'audio',
|
||||
characters: ['Ethan'],
|
||||
description: 'Adversarial lookalike: a podcast episode discussing pollen forecasts.',
|
||||
timestamp: '2026-05-11T09:30:00',
|
||||
location: 'kitchen',
|
||||
source: 'adversarial',
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
questions: [
|
||||
{
|
||||
question_id: 'q1',
|
||||
question: 'Why did Sarah change the purifier and curtains for pollen?',
|
||||
answer: 'Because of pollen.',
|
||||
answer_checklist: [
|
||||
{ point: 'purifier to turbo', source_events: ['sc1_ep1_e2'] },
|
||||
{ point: 'curtains closed', source_events: ['sc1_ep1_e3'] },
|
||||
],
|
||||
evidence_event_ids: ['sc1_ep1_e2', 'sc1_ep1_e3'],
|
||||
dimension: 'cross_device_causal_reasoning',
|
||||
question_user: 'David',
|
||||
question_time: '2026-05-11T12:30:00',
|
||||
question_device: 'phone_david',
|
||||
},
|
||||
{
|
||||
question_id: 'q2',
|
||||
question: 'What was the living room thermostat temperature reading?',
|
||||
answer: '21.8C',
|
||||
answer_checklist: [{ point: 'temperature', source_events: ['sc1_ep1_e1'] }],
|
||||
evidence_event_ids: ['sc1_ep1_e1'],
|
||||
dimension: 'cross_device_information_fusion',
|
||||
question_user: 'Sarah',
|
||||
question_time: '2026-05-11T13:00:00',
|
||||
question_device: 'tablet_home',
|
||||
},
|
||||
],
|
||||
},
|
||||
{
|
||||
scenario_id: 'sc2',
|
||||
episodes: [
|
||||
{
|
||||
episode_id: 'sc2_ep1',
|
||||
events: [
|
||||
{
|
||||
event_id: 'sc2_ep1_e1',
|
||||
device: 'watch_ethan',
|
||||
modality: 'wearable',
|
||||
characters: ['Ethan'],
|
||||
description: 'Wearable recorded an elevated heart rate during the afternoon run.',
|
||||
timestamp: '2026-05-12T17:05:00',
|
||||
location: 'park',
|
||||
source: 'storyline',
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
questions: [
|
||||
{
|
||||
question_id: 'q1',
|
||||
question: 'What did the wearable record during the afternoon run?',
|
||||
answer: 'Elevated heart rate.',
|
||||
answer_checklist: [{ point: 'elevated heart rate', source_events: ['sc2_ep1_e1'] }],
|
||||
evidence_event_ids: ['sc2_ep1_e1'],
|
||||
dimension: 'perspective_difference',
|
||||
question_user: 'Sarah',
|
||||
question_time: '2026-05-12T19:00:00',
|
||||
question_device: 'phone_sarah',
|
||||
},
|
||||
],
|
||||
},
|
||||
],
|
||||
};
|
||||
}
|
||||
|
||||
function loadFixtureDataset() {
|
||||
const normalized = normalizeMemFuseDataset(makeRawDataset());
|
||||
assert.equal(normalized.ok, true);
|
||||
return normalized.dataset;
|
||||
}
|
||||
|
||||
test('resolveMemFuseDatasetPath prefers the explicit env override', () => {
|
||||
const explicit = resolveMemFuseDatasetPath({ [MEMFUSE_DATASET_ENV]: '/tmp/custom/bench.json' });
|
||||
assert.equal(explicit, '/tmp/custom/bench.json');
|
||||
|
||||
const viaRoot = resolveMemFuseDatasetPath({ MEMFUSE_PROJECT_ROOT: '/srv/refs' });
|
||||
assert.equal(viaRoot, '/srv/refs/mi-memory/MemFuse/MemFuseBench/memfusebench_dataset.json');
|
||||
});
|
||||
|
||||
test('normalizeMemFuseDataset keeps usable scenarios and reports stats', () => {
|
||||
const normalized = normalizeMemFuseDataset(makeRawDataset());
|
||||
assert.equal(normalized.ok, true);
|
||||
assert.deepEqual(normalized.dataset.stats, {
|
||||
scenarioCount: 2,
|
||||
eventCount: 6,
|
||||
questionCount: 3,
|
||||
});
|
||||
const [scenario] = normalized.dataset.scenarios;
|
||||
assert.equal(scenario.scenarioId, 'sc1');
|
||||
assert.equal(scenario.questions[0].evidenceEventIds.length, 2);
|
||||
assert.equal(scenario.questions[0].checklist.length, 2);
|
||||
});
|
||||
|
||||
test('normalizeMemFuseDataset drops events and questions that cannot be scored', () => {
|
||||
const raw = makeRawDataset();
|
||||
raw.scenarios[0].episodes[0].events.push({ event_id: 'no_description' });
|
||||
raw.scenarios[0].questions.push({
|
||||
question_id: 'q_no_evidence',
|
||||
question: 'Unscoreable question',
|
||||
evidence_event_ids: [],
|
||||
});
|
||||
const normalized = normalizeMemFuseDataset(raw);
|
||||
assert.equal(normalized.ok, true);
|
||||
assert.equal(normalized.dataset.stats.eventCount, 6);
|
||||
assert.equal(normalized.dataset.stats.questionCount, 3);
|
||||
});
|
||||
|
||||
test('normalizeMemFuseDataset rejects payloads with no usable scenario', () => {
|
||||
assert.equal(normalizeMemFuseDataset({}).reason, 'dataset_has_no_usable_scenarios');
|
||||
assert.equal(normalizeMemFuseDataset({ scenarios: [{}] }).reason, 'dataset_has_no_usable_scenarios');
|
||||
});
|
||||
|
||||
test('loadMemFuseDataset degrades instead of throwing when the dataset is absent', async () => {
|
||||
const missing = await loadMemFuseDataset({
|
||||
datasetPath: '/tmp/definitely-absent.json',
|
||||
readFile: async () => {
|
||||
const err = new Error('ENOENT');
|
||||
err.code = 'ENOENT';
|
||||
throw err;
|
||||
},
|
||||
});
|
||||
assert.equal(missing.available, false);
|
||||
assert.equal(missing.reason, 'dataset_not_found');
|
||||
|
||||
const unparsable = await loadMemFuseDataset({
|
||||
datasetPath: '/tmp/broken.json',
|
||||
readFile: async () => 'not json',
|
||||
});
|
||||
assert.equal(unparsable.available, false);
|
||||
assert.equal(unparsable.reason, 'dataset_parse_failed');
|
||||
});
|
||||
|
||||
test('loadMemFuseDataset returns normalized scenarios on success', async () => {
|
||||
const loaded = await loadMemFuseDataset({
|
||||
datasetPath: '/tmp/fixture.json',
|
||||
readFile: async () => JSON.stringify(makeRawDataset()),
|
||||
});
|
||||
assert.equal(loaded.available, true);
|
||||
assert.equal(loaded.path, '/tmp/fixture.json');
|
||||
assert.equal(loaded.stats.questionCount, 3);
|
||||
});
|
||||
|
||||
test('buildScenarioCorpus emits pgvector-shaped rows and can prepend source tags', () => {
|
||||
const [scenario] = loadFixtureDataset().scenarios;
|
||||
const plain = buildScenarioCorpus(scenario);
|
||||
assert.equal(plain.rows.length, 5);
|
||||
const first = plain.byId.get('sc1_ep1_e1');
|
||||
assert.equal(first.content, 'Living room thermostat reading: temperature 21.8C, humidity 47%.');
|
||||
assert.equal(first.type, 'environment');
|
||||
assert.equal(first.source, 'periodic');
|
||||
assert.equal(first.episodeId, 'sc1_ep1');
|
||||
assert.equal(first.created_at, new Date('2026-05-11T06:00:00').toISOString());
|
||||
|
||||
const tagged = buildScenarioCorpus(scenario, { includeSourceTags: true });
|
||||
assert.equal(
|
||||
tagged.byId.get('sc1_ep1_e1').content,
|
||||
'[thermostat_living · living_room] Living room thermostat reading: temperature 21.8C, humidity 47%.',
|
||||
);
|
||||
});
|
||||
|
||||
test('buildScenarioCases filters by dimension, id and cap', () => {
|
||||
const [scenario] = loadFixtureDataset().scenarios;
|
||||
assert.equal(buildScenarioCases(scenario).length, 2);
|
||||
assert.deepEqual(
|
||||
buildScenarioCases(scenario, { dimensions: ['cross_device_causal_reasoning'] }).map(
|
||||
(item) => item.questionId,
|
||||
),
|
||||
['q1'],
|
||||
);
|
||||
assert.deepEqual(
|
||||
buildScenarioCases(scenario, { questionIds: ['q2'] }).map((item) => item.questionId),
|
||||
['q2'],
|
||||
);
|
||||
assert.equal(buildScenarioCases(scenario, { maxQuestions: 1 }).length, 1);
|
||||
});
|
||||
|
||||
test('createLexicalHashEmbedder is deterministic and L2 normalized', () => {
|
||||
const embed = createLexicalHashEmbedder({ dimensions: 64 });
|
||||
const left = embed('pollen purifier curtains');
|
||||
const right = embed('pollen purifier curtains');
|
||||
assert.deepEqual(left, right);
|
||||
assert.equal(left.length, 64);
|
||||
const norm = Math.sqrt(left.reduce((total, value) => total + value * value, 0));
|
||||
assert.ok(Math.abs(norm - 1) < 1e-9);
|
||||
assert.notDeepEqual(embed('completely different text'), left);
|
||||
assert.deepEqual(
|
||||
embed(''),
|
||||
new Array(64).fill(0),
|
||||
);
|
||||
});
|
||||
|
||||
test('createCorpusPool emulates the vector union and the ILIKE fallback', async () => {
|
||||
const [scenario] = loadFixtureDataset().scenarios;
|
||||
const corpus = buildScenarioCorpus(scenario);
|
||||
const embed = createLexicalHashEmbedder({ dimensions: 128 });
|
||||
const pool = createCorpusPool({ rows: corpus.rows, embedText: embed });
|
||||
|
||||
const vectorSql = 'WITH vector_candidates AS (SELECT ...) SELECT DISTINCT ON (id) ...';
|
||||
const vectorResult = await pool.query(vectorSql, [
|
||||
'user',
|
||||
`[${embed('pollen purifier').join(',')}]`,
|
||||
3,
|
||||
]);
|
||||
assert.ok(vectorResult.rows.length > 0);
|
||||
assert.ok(vectorResult.rows.every((row) => typeof row.score === 'number'));
|
||||
assert.ok(vectorResult.rows.some((row) => row.id === 'sc1_ep1_e2'));
|
||||
|
||||
const keywordResult = await pool.query(
|
||||
'SELECT id, content FROM memory_embeddings WHERE user_id = $1 AND (content ILIKE $2) LIMIT $3',
|
||||
['user', '%curtains%', 10],
|
||||
);
|
||||
assert.deepEqual(
|
||||
keywordResult.rows.map((row) => row.id),
|
||||
['sc1_ep1_e3'],
|
||||
);
|
||||
|
||||
const unknown = await pool.query('SELECT 1', []);
|
||||
assert.deepEqual(unknown.rows, []);
|
||||
assert.equal(pool.stats().corpusSize, 5);
|
||||
assert.equal(pool.stats().vectorQueryCount, 1);
|
||||
assert.equal(pool.stats().keywordQueryCount, 1);
|
||||
});
|
||||
|
||||
test('runMemFuseBenchCase scores recall, checklist coverage and distractors', async () => {
|
||||
const [scenario] = loadFixtureDataset().scenarios;
|
||||
const corpus = buildScenarioCorpus(scenario);
|
||||
const [testCase] = scenario.questions;
|
||||
const result = await runMemFuseBenchCase({
|
||||
testCase,
|
||||
corpus,
|
||||
embedText: createLexicalHashEmbedder({ dimensions: 256 }),
|
||||
limit: 5,
|
||||
});
|
||||
|
||||
assert.equal(result.questionId, 'q1');
|
||||
assert.equal(result.dimension, 'cross_device_causal_reasoning');
|
||||
assert.equal(result.goldCount, 2);
|
||||
assert.equal(result.hitCount, 2);
|
||||
assert.equal(result.recall, 1);
|
||||
assert.equal(result.hitAny, true);
|
||||
assert.equal(result.checklistCoverage, 1);
|
||||
assert.ok(result.firstHitRank >= 1);
|
||||
assert.ok(result.reciprocalRank > 0);
|
||||
assert.ok(result.precision > 0 && result.precision <= 1);
|
||||
// e4 (noise) and e5 (adversarial) are the only planted distractors.
|
||||
assert.ok(result.distractorCount <= 2);
|
||||
// Small corpus: every row reaches the candidate pool, so ranking loses nothing.
|
||||
assert.equal(result.candidateRecall, 1);
|
||||
assert.equal(result.rankingLoss, 0);
|
||||
assert.equal(result.candidateCount, 5);
|
||||
});
|
||||
|
||||
test('runMemFuseBenchCase separates candidate recall from ranking loss', async () => {
|
||||
const [scenario] = loadFixtureDataset().scenarios;
|
||||
const corpus = buildScenarioCorpus(scenario);
|
||||
const [testCase] = scenario.questions;
|
||||
// limit 1 forces ranking to drop evidence that candidate generation found.
|
||||
const result = await runMemFuseBenchCase({
|
||||
testCase,
|
||||
corpus,
|
||||
embedText: createLexicalHashEmbedder({ dimensions: 256 }),
|
||||
limit: 1,
|
||||
});
|
||||
assert.equal(result.candidateRecall, 1);
|
||||
assert.ok(result.recall < 1);
|
||||
assert.ok(result.rankingLoss > 0);
|
||||
assert.equal(result.rankingLoss, result.candidateRecall - result.recall);
|
||||
});
|
||||
|
||||
test('runMemFuseBenchCase reports a clean miss when nothing relevant exists', async () => {
|
||||
const [scenario] = loadFixtureDataset().scenarios;
|
||||
const corpus = buildScenarioCorpus(scenario);
|
||||
const result = await runMemFuseBenchCase({
|
||||
testCase: {
|
||||
questionId: 'synthetic-miss',
|
||||
scenarioId: 'sc1',
|
||||
dimension: 'cross_user_query',
|
||||
question: 'zzz nonexistent topic zzz',
|
||||
evidenceEventIds: ['sc1_ep1_missing'],
|
||||
checklist: [{ point: 'absent', sourceEvents: ['sc1_ep1_missing'] }],
|
||||
},
|
||||
corpus,
|
||||
embedText: createLexicalHashEmbedder({ dimensions: 256 }),
|
||||
limit: 5,
|
||||
});
|
||||
assert.equal(result.hitCount, 0);
|
||||
assert.equal(result.recall, 0);
|
||||
assert.equal(result.hitAny, false);
|
||||
assert.equal(result.firstHitRank, null);
|
||||
assert.equal(result.reciprocalRank, 0);
|
||||
assert.equal(result.checklistCoverage, 0);
|
||||
});
|
||||
|
||||
test('runMemFuseBench aggregates overall, per-scenario and per-dimension metrics', async () => {
|
||||
const dataset = loadFixtureDataset();
|
||||
const report = await runMemFuseBench({ dataset, limit: 5 });
|
||||
|
||||
assert.equal(report.embeddingMode, 'lexical-hash');
|
||||
assert.equal(report.limit, 5);
|
||||
assert.equal(report.overall.caseCount, 3);
|
||||
assert.ok(report.overall.recallAtK > 0);
|
||||
assert.ok(report.overall.hitAnyRate > 0);
|
||||
assert.deepEqual(
|
||||
report.byScenario.map((item) => item.scenarioId),
|
||||
['sc1', 'sc2'],
|
||||
);
|
||||
assert.equal(report.byScenario[0].corpusSize, 5);
|
||||
assert.ok(Object.keys(report.byDimension).length >= 2);
|
||||
assert.equal(report.results.length, 3);
|
||||
});
|
||||
|
||||
test('runMemFuseBench honours scenario, dimension and cap filters', async () => {
|
||||
const dataset = loadFixtureDataset();
|
||||
const report = await runMemFuseBench({
|
||||
dataset,
|
||||
scenarioIds: ['sc1'],
|
||||
dimensions: ['cross_device_information_fusion'],
|
||||
limit: 5,
|
||||
});
|
||||
assert.equal(report.overall.caseCount, 1);
|
||||
assert.equal(report.results[0].questionId, 'q2');
|
||||
|
||||
const capped = await runMemFuseBench({ dataset, maxQuestionsPerScenario: 1, limit: 5 });
|
||||
assert.equal(capped.overall.caseCount, 2);
|
||||
});
|
||||
|
||||
test('runMemFuseBench clamps limit to the pgvector backend ceiling', async () => {
|
||||
const report = await runMemFuseBench({ dataset: loadFixtureDataset(), limit: 500 });
|
||||
assert.equal(report.limit, 50);
|
||||
});
|
||||
|
||||
test('runMemFuseBench marks an injected embedder as external and reports progress', async () => {
|
||||
const dataset = loadFixtureDataset();
|
||||
const seen = [];
|
||||
const report = await runMemFuseBench({
|
||||
dataset,
|
||||
limit: 5,
|
||||
embedText: createLexicalHashEmbedder({ dimensions: 32 }),
|
||||
onProgress: (event) => seen.push(event.questionId),
|
||||
});
|
||||
assert.equal(report.embeddingMode, 'external');
|
||||
assert.equal(seen.length, 3);
|
||||
});
|
||||
|
||||
test('runMemFuseBench rejects an empty dataset or an empty selection', async () => {
|
||||
await assert.rejects(() => runMemFuseBench({ dataset: { scenarios: [] } }), /requires a loaded dataset/);
|
||||
await assert.rejects(
|
||||
() => runMemFuseBench({ dataset: loadFixtureDataset(), dimensions: ['not_a_dimension'] }),
|
||||
/selected zero questions/,
|
||||
);
|
||||
});
|
||||
|
||||
test('summarizeMemFuseBench rounds metrics and names the weakest dimension', async () => {
|
||||
const report = await runMemFuseBench({ dataset: loadFixtureDataset(), limit: 5 });
|
||||
const summary = summarizeMemFuseBench(report);
|
||||
assert.equal(summary.caseCount, 3);
|
||||
assert.equal(summary.embeddingMode, 'lexical-hash');
|
||||
assert.ok(summary.recallAtK >= 0 && summary.recallAtK <= 1);
|
||||
assert.ok(summary.candidateRecall >= summary.recallAtK);
|
||||
assert.ok(summary.rankingLoss >= 0);
|
||||
assert.ok(summary.mrr >= 0 && summary.mrr <= 1);
|
||||
assert.ok(summary.distractorRate >= 0 && summary.distractorRate <= 1);
|
||||
assert.ok(typeof summary.weakestDimension === 'string');
|
||||
});
|
||||
@@ -69,6 +69,8 @@
|
||||
"repair:memory-v2-candidates": "node scripts/repair-memory-v2-candidates.mjs",
|
||||
"verify:memory-v2-shadow-audit": "node --test memory-v2-shadow-audit.test.mjs",
|
||||
"verify:memory-v2-recall-benchmark": "node --test memory-v2-recall-benchmark.test.mjs",
|
||||
"verify:memory-v2-memfuse-bench": "node --test memory-v2-memfuse-bench.test.mjs",
|
||||
"bench:memory-v2-memfuse": "node scripts/run-memory-v2-memfuse-bench.mjs",
|
||||
"trace:mindspace-artifact": "node scripts/trace-mindspace-artifact.mjs",
|
||||
"check:conversation-package-manifest": "node scripts/check-conversation-package-manifest.mjs",
|
||||
"check:memory-v2-phase-a": "node scripts/check-memory-v2-phase-a-ready.mjs",
|
||||
|
||||
@@ -0,0 +1,242 @@
|
||||
#!/usr/bin/env node
|
||||
/**
|
||||
* Run the MemFuseBench retrieval benchmark against the Memory V2 pgvector
|
||||
* ranking path.
|
||||
*
|
||||
* The dataset lives outside this repository (MIT, Mi-Memory / Darwin Agent
|
||||
* Team). Point at it with --dataset or MEMFUSE_BENCH_DATASET; without it the
|
||||
* run reports `skipped` and exits 0 unless --strict is passed.
|
||||
*
|
||||
* Usage:
|
||||
* node scripts/run-memory-v2-memfuse-bench.mjs
|
||||
* node scripts/run-memory-v2-memfuse-bench.mjs --scenario sc1 --limit 20
|
||||
* node scripts/run-memory-v2-memfuse-bench.mjs --dimension multi_source_conflict_arbitration
|
||||
* node scripts/run-memory-v2-memfuse-bench.mjs --embedding-module ./scripts/my-embedder.mjs
|
||||
* node scripts/run-memory-v2-memfuse-bench.mjs --json --out .release-gate/memfuse.json
|
||||
*/
|
||||
|
||||
import fs from 'node:fs/promises';
|
||||
import path from 'node:path';
|
||||
|
||||
import {
|
||||
MEMFUSE_DIMENSIONS,
|
||||
loadMemFuseDataset,
|
||||
runMemFuseBench,
|
||||
summarizeMemFuseBench,
|
||||
} from '../memory-v2-memfuse-bench.mjs';
|
||||
import { resolveEmbeddingModuleSpecifier } from '../memory-v2-recall-benchmark.mjs';
|
||||
|
||||
function parseArgs(argv) {
|
||||
const options = {
|
||||
dataset: null,
|
||||
scenarios: null,
|
||||
dimensions: null,
|
||||
questionIds: null,
|
||||
limit: 20,
|
||||
candidateLimit: 100,
|
||||
maxQuestions: null,
|
||||
sourceTags: false,
|
||||
embeddingModule: null,
|
||||
json: false,
|
||||
out: null,
|
||||
strict: false,
|
||||
quiet: false,
|
||||
};
|
||||
const list = (value) =>
|
||||
String(value ?? '')
|
||||
.split(',')
|
||||
.map((item) => item.trim())
|
||||
.filter(Boolean);
|
||||
|
||||
for (let index = 0; index < argv.length; index += 1) {
|
||||
const arg = argv[index];
|
||||
const next = () => argv[(index += 1)];
|
||||
switch (arg) {
|
||||
case '--dataset': options.dataset = next(); break;
|
||||
case '--scenario': options.scenarios = list(next()); break;
|
||||
case '--dimension': options.dimensions = list(next()); break;
|
||||
case '--question': options.questionIds = list(next()); break;
|
||||
case '--limit': options.limit = Number(next()); break;
|
||||
case '--candidate-limit': options.candidateLimit = Number(next()); break;
|
||||
case '--max-questions': options.maxQuestions = Number(next()); break;
|
||||
case '--source-tags': options.sourceTags = true; break;
|
||||
case '--embedding-module': options.embeddingModule = next(); break;
|
||||
case '--json': options.json = true; break;
|
||||
case '--out': options.out = next(); break;
|
||||
case '--strict': options.strict = true; break;
|
||||
case '--quiet': options.quiet = true; break;
|
||||
case '-h':
|
||||
case '--help': options.help = true; break;
|
||||
default:
|
||||
if (arg.startsWith('--')) throw new Error(`Unknown flag: ${arg}`);
|
||||
}
|
||||
}
|
||||
return options;
|
||||
}
|
||||
|
||||
function usage() {
|
||||
process.stdout.write(`Usage: node scripts/run-memory-v2-memfuse-bench.mjs [options]
|
||||
|
||||
--dataset <path> MemFuseBench dataset JSON (default: $MEMFUSE_BENCH_DATASET)
|
||||
--scenario <ids> Comma-separated scenario ids, e.g. sc1,sc2
|
||||
--dimension <names> Comma-separated dimensions. Known values:
|
||||
${MEMFUSE_DIMENSIONS.join('\n ')}
|
||||
--question <ids> Comma-separated question ids
|
||||
--limit <n> Retrieved rows per question (clamped to 50)
|
||||
--candidate-limit <n> Candidate pool size before ranking (clamped to 100)
|
||||
--max-questions <n> Cap questions per scenario (useful for smoke runs)
|
||||
--source-tags Prepend "[device · location]" to event text
|
||||
--embedding-module <spec> Module exporting embedQuery/embedText/default
|
||||
--json Print the full report as JSON
|
||||
--out <path> Write the JSON report to a file
|
||||
--strict Exit non-zero when the dataset is unavailable
|
||||
--quiet Suppress per-scenario progress
|
||||
`);
|
||||
}
|
||||
|
||||
async function loadEmbedder(specifier) {
|
||||
const resolved = resolveEmbeddingModuleSpecifier(specifier);
|
||||
if (!resolved) throw new Error(`Cannot resolve embedding module: ${specifier}`);
|
||||
const imported = await import(resolved);
|
||||
const embed = imported?.embedText ?? imported?.embedQuery ?? imported?.default;
|
||||
if (typeof embed !== 'function') {
|
||||
throw new Error(`Embedding module must export embedText, embedQuery or default: ${specifier}`);
|
||||
}
|
||||
return embed;
|
||||
}
|
||||
|
||||
function formatPercent(value) {
|
||||
return Number.isFinite(value) ? `${(value * 100).toFixed(1)}%` : 'n/a';
|
||||
}
|
||||
|
||||
function formatRow(label, metrics, width) {
|
||||
return [
|
||||
label.padEnd(width),
|
||||
String(metrics.caseCount).padStart(5),
|
||||
formatPercent(metrics.candidateRecall).padStart(9),
|
||||
formatPercent(metrics.recallAtK).padStart(8),
|
||||
formatPercent(metrics.rankingLoss).padStart(8),
|
||||
formatPercent(metrics.hitAnyRate).padStart(8),
|
||||
formatPercent(metrics.checklistCoverage).padStart(10),
|
||||
(Number.isFinite(metrics.mrr) ? metrics.mrr.toFixed(3) : 'n/a').padStart(7),
|
||||
formatPercent(metrics.distractorRate).padStart(11),
|
||||
].join(' ');
|
||||
}
|
||||
|
||||
function printReport(report) {
|
||||
const labels = [
|
||||
...report.byScenario.map((item) => item.scenarioId),
|
||||
...Object.keys(report.byDimension),
|
||||
'OVERALL',
|
||||
];
|
||||
const width = Math.max(12, ...labels.map((label) => label.length));
|
||||
const header = [
|
||||
'group'.padEnd(width),
|
||||
'cases'.padStart(5),
|
||||
'cand.rec'.padStart(9),
|
||||
'recall'.padStart(8),
|
||||
'rankLoss'.padStart(8),
|
||||
'hit@k'.padStart(8),
|
||||
'checklist'.padStart(10),
|
||||
'mrr'.padStart(7),
|
||||
'distractor'.padStart(11),
|
||||
].join(' ');
|
||||
|
||||
process.stdout.write(`\nMemFuseBench retrieval @k=${report.limit}`);
|
||||
process.stdout.write(
|
||||
` · embedding=${report.embeddingMode}` +
|
||||
` · sourceTags=${report.includeSourceTags ? 'on' : 'off'}\n`,
|
||||
);
|
||||
if (report.embeddingMode === 'lexical-hash') {
|
||||
process.stdout.write(
|
||||
'NOTE: lexical-hash is a deterministic offline stand-in, not a semantic embedder.\n' +
|
||||
' Use --embedding-module to measure real semantic recall.\n',
|
||||
);
|
||||
}
|
||||
process.stdout.write(`\n${header}\n${'-'.repeat(header.length)}\n`);
|
||||
for (const scenario of report.byScenario) {
|
||||
process.stdout.write(`${formatRow(scenario.scenarioId, scenario, width)}\n`);
|
||||
}
|
||||
process.stdout.write(`${'-'.repeat(header.length)}\n`);
|
||||
for (const [dimension, metrics] of Object.entries(report.byDimension).sort(
|
||||
(left, right) => left[1].recallAtK - right[1].recallAtK,
|
||||
)) {
|
||||
process.stdout.write(`${formatRow(dimension, metrics, width)}\n`);
|
||||
}
|
||||
process.stdout.write(`${'-'.repeat(header.length)}\n`);
|
||||
process.stdout.write(`${formatRow('OVERALL', report.overall, width)}\n\n`);
|
||||
}
|
||||
|
||||
async function main() {
|
||||
const options = parseArgs(process.argv.slice(2));
|
||||
if (options.help) {
|
||||
usage();
|
||||
return 0;
|
||||
}
|
||||
|
||||
const loaded = await loadMemFuseDataset({ datasetPath: options.dataset });
|
||||
if (!loaded.available) {
|
||||
const message =
|
||||
`MemFuseBench dataset unavailable (${loaded.reason}) at ${loaded.path}\n` +
|
||||
'Clone it outside this repo, then set MEMFUSE_BENCH_DATASET or pass --dataset:\n' +
|
||||
' git clone https://github.com/Darwin-Agent/Mi-Memory ~/Project/mi-memory\n';
|
||||
process.stderr.write(message);
|
||||
return options.strict ? 1 : 0;
|
||||
}
|
||||
|
||||
if (!options.quiet) {
|
||||
process.stdout.write(
|
||||
`dataset: ${loaded.path}\n` +
|
||||
`scenarios=${loaded.stats.scenarioCount} ` +
|
||||
`events=${loaded.stats.eventCount} questions=${loaded.stats.questionCount}\n`,
|
||||
);
|
||||
}
|
||||
|
||||
const embedText = options.embeddingModule
|
||||
? await loadEmbedder(options.embeddingModule)
|
||||
: null;
|
||||
|
||||
let lastScenario = null;
|
||||
const report = await runMemFuseBench({
|
||||
dataset: loaded,
|
||||
scenarioIds: options.scenarios,
|
||||
dimensions: options.dimensions,
|
||||
questionIds: options.questionIds,
|
||||
maxQuestionsPerScenario: Number.isFinite(options.maxQuestions) ? options.maxQuestions : null,
|
||||
limit: options.limit,
|
||||
candidateLimit: options.candidateLimit,
|
||||
includeSourceTags: options.sourceTags,
|
||||
embedText,
|
||||
onProgress: options.quiet
|
||||
? null
|
||||
: (event) => {
|
||||
if (event.scenarioId !== lastScenario) {
|
||||
lastScenario = event.scenarioId;
|
||||
process.stderr.write(` running ${event.scenarioId}...\n`);
|
||||
}
|
||||
},
|
||||
});
|
||||
|
||||
if (options.json) {
|
||||
process.stdout.write(`${JSON.stringify(report, null, 2)}\n`);
|
||||
} else {
|
||||
printReport(report);
|
||||
process.stdout.write(`${JSON.stringify(summarizeMemFuseBench(report), null, 2)}\n`);
|
||||
}
|
||||
|
||||
if (options.out) {
|
||||
const target = path.resolve(options.out);
|
||||
await fs.mkdir(path.dirname(target), { recursive: true });
|
||||
await fs.writeFile(target, `${JSON.stringify(report, null, 2)}\n`, 'utf8');
|
||||
process.stderr.write(`report written to ${target}\n`);
|
||||
}
|
||||
|
||||
return 0;
|
||||
}
|
||||
|
||||
main()
|
||||
.then((code) => process.exit(code))
|
||||
.catch((err) => {
|
||||
process.stderr.write(`${err instanceof Error ? err.stack : String(err)}\n`);
|
||||
process.exit(1);
|
||||
});
|
||||
Reference in New Issue
Block a user