memind/docs/memory-v2/production-rollout-runbook.md

# Memory V2 Production Rollout Runbook

Memory V2 is a facade and policy layer. Production rollout must not introduce new goosed, PG session, SSE, or Portal execution paths.

## Release Gates

Run the normal repository release checks first:

```bash
bash scripts/check-release-ready.sh
```

Run the Memory V2 backend contract gate:

```bash
npm run check:memory-v2-contracts
```

Run the Memory V2 health gate:

```bash
npm run check:memory-v2 -- --require-enabled --expect-backend legacy
```

Expected phase-one result:

- `ok=true`
- contract gate reports unique backend names and legacy backend present
- `summary.enabled=true`
- `summary.backend=legacy`
- `summary.selectedBackend=legacy-conversation-memory`
- `fail_open` passes
- `legacy_write_compact_supported` passes
- `write_uses_legacy` passes
- `compact_uses_legacy` passes

When validating the real pgvector adapter before canary:

```bash
npm run check:memory-v2-contracts -- --include-pgvector-adapter
```

Future backend adapter rule:

```bash
npm run scaffold:memory-v2-backend -- \
  --name qdrant \
  --category semantic \
  --capability resolve
```

Use the scaffold as the starting point, add focused adapter tests, then pass `npm run check:memory-v2-contracts` before wiring any external SDK into runtime. Do not wire external services directly into `server.mjs`, `tkmind-proxy.mjs`, goosed, SSE, or Portal execution code.

## Qdrant Read-Only Phase

The Qdrant adapter may be visible in runtime status before user traffic is canaried.

Fallback-only environment:

```bash
MEMORY_ENABLED=1
MEMORY_BACKEND=qdrant
MEMORY_QDRANT_ENABLED=1
MEMORY_QDRANT_URL='http://qdrant.example:6333'
MEMORY_QDRANT_COLLECTION='memind_memory'
```

Expected without embedding module:

- `memory.backend=qdrant`
- `memory.selectedBackend=legacy-conversation-memory`
- `qdrant.available=false`
- `qdrant.reason=embedding_module_not_configured`
- `write_uses_legacy` still passes
- `compact_uses_legacy` still passes

Read-only smoke:

```bash
MEMORY_QDRANT_URL='http://qdrant.example:6333' \
npm run smoke:memory-v2-qdrant -- \
  --collection memind_memory \
  --vector 1,0,0 \
  --limit 1
```

The smoke command sends a Qdrant search request only. It must not create collections, write points, or run from app startup.

Runtime read-only canary requires:

```bash
MEMORY_QDRANT_EMBEDDING_MODULE='./scripts/embed-memory-v2-qdrant.mjs'
```

Even during read-only canary, `write_uses_legacy` and `compact_uses_legacy` must still pass.

## Mem0 Adapter Phase

Mem0 is the Memory Extraction backend boundary. The adapter uses HTTP and is dormant unless explicitly configured.

Environment:

```bash
MEMORY_ENABLED=1
MEMORY_BACKEND=mem0
MEMORY_MEM0_ENABLED=1
MEMORY_MEM0_API_KEY='...'
MEMORY_MEM0_PROJECT_ID='memind_project'
MEMORY_MEM0_BASE_URL='https://api.mem0.ai'
```

Smoke:

```bash
npm run smoke:memory-v2-external -- --backend mem0 --operation write
```

Do not canary Mem0 writes until sampling, dedupe, rollback, and retention rules are accepted.

## Letta Lifecycle Adapter Phase

Letta is the Memory Lifecycle Management backend boundary. The adapter uses HTTP and is dormant unless explicitly configured.

Environment:

```bash
MEMORY_ENABLED=1
MEMORY_BACKEND=letta
MEMORY_LETTA_ENABLED=1
MEMORY_LETTA_API_KEY='...'
MEMORY_LETTA_PROJECT_ID='memind_project'
MEMORY_LETTA_AGENT_ID='agent_1'
MEMORY_LETTA_BASE_URL='https://api.letta.com'
```

Smoke:

```bash
npm run smoke:memory-v2-external -- --backend letta --operation resolve
```

Do not canary Letta until ownership boundaries, long/short-term promotion rules, conflict resolution with conversation-memory, rollback, and retention are accepted.

## Remaining External Adapter Phase

Weaviate, Neo4j, Redis Streams, and LangGraph are wired with runtime clients and remain dormant unless explicitly configured.

Environment examples:

```bash
MEMORY_WEAVIATE_ENABLED=1
MEMORY_WEAVIATE_URL='https://weaviate.example'
MEMORY_WEAVIATE_COLLECTION='MemindMemory'
MEMORY_WEAVIATE_EMBEDDING_MODULE='./scripts/embed-memory-v2-weaviate.mjs'

MEMORY_NEO4J_ENABLED=1
MEMORY_NEO4J_HTTP_URL='http://localhost:7474'
MEMORY_NEO4J_USER='neo4j'
MEMORY_NEO4J_PASSWORD='...'

MEMORY_REDIS_STREAMS_ENABLED=1
MEMORY_REDIS_STREAMS_URL='redis://localhost:6379'

MEMORY_LANGGRAPH_ENABLED=1
MEMORY_LANGGRAPH_URL='https://langgraph.example'
MEMORY_LANGGRAPH_POLICY_ID='memory_policy'
```

Smoke:

```bash
npm run smoke:memory-v2-external -- --backend weaviate --operation resolve
npm run smoke:memory-v2-external -- --backend neo4j --operation write
npm run smoke:memory-v2-external -- --backend redis-streams --operation write
npm run smoke:memory-v2-external -- --backend langgraph --operation resolve
```

Future canaries must be backend-specific and must define data ownership, dedupe, observability, rollback, and retention before any adapter can own a Memory V2 operation.

## Phase 1: Facade Over Legacy

Environment:

```bash
MEMORY_ENABLED=1
MEMORY_BACKEND=legacy
MEMORY_FAIL_OPEN=1
MEMORY_VECTOR_ENABLED=0
```

Runtime verification:

```bash
curl -sk https://<host>/api/runtime/status
```

Confirm:

- `memory.enabled=true`
- `memory.backend=legacy`
- `memory.selectedBackend=legacy-conversation-memory`
- future plugin slots report `available=false`

Rollback:

```bash
MEMORY_ENABLED=0
```

## Phase 2: Empty pgvector Schema

This phase creates schema only. It must not change runtime backend selection.

Preconditions:

- PostgreSQL server is approved for Memory V2 semantic memory.
- `pgvector` extension is available.
- Backup/rollback decision is recorded by ops.

Dry-run:

```bash
node scripts/setup-memory-v2-pgvector-schema.mjs \
  --table memory_embeddings \
  --dimensions 1536 \
  --create-extension
```

Apply only through the approved production migration path:

```bash
MEMORY_PGVECTOR_DATABASE_URL='postgresql://...' \
node scripts/setup-memory-v2-pgvector-schema.mjs \
  --apply \
  --table memory_embeddings \
  --dimensions 1536 \
  --create-extension
```

Keep runtime on legacy:

```bash
MEMORY_BACKEND=legacy
MEMORY_VECTOR_ENABLED=0
```

Post-migration smoke verification:

```bash
MEMORY_PGVECTOR_DATABASE_URL='postgresql://...' \
npm run smoke:memory-v2-pgvector -- \
  --table memory_embeddings \
  --dimensions 1536
```

The smoke command inserts synthetic `memory-v2-smoke-*` rows, verifies nearest-vector retrieval, and deletes those rows by default. In production it should run only after the schema migration. Do not pass `--create-schema` in production unless the approved migration procedure explicitly requires it.

For local-only plumbing checks, use the deterministic hash embedding module:

```bash
MEMORY_PGVECTOR_EMBEDDING_MODULE=./scripts/embed-memory-v2-local-hash.mjs
MEMORY_V2_LOCAL_EMBEDDING_DIMENSIONS=3
```

Do not use this local hash embedding module for production retrieval quality.

## Phase 3: Backfill

Backfill must be separate from schema creation and application rollout.

Dry-run:

```bash
MEMORY_BACKFILL_MYSQL_URL='mysql://...' \
node scripts/backfill-memory-v2-pgvector.mjs --limit 100
```

Apply:

```bash
MEMORY_BACKFILL_MYSQL_URL='mysql://...' \
MEMORY_PGVECTOR_DATABASE_URL='postgresql://...' \
node scripts/backfill-memory-v2-pgvector.mjs \
  --apply \
  --limit 100 \
  --embedding-module ./scripts/embed-memory-v2-pgvector.mjs
```

Operational rules:

- Store `{ updatedAt, id }` checkpoint externally after each successful batch.
- Compare active MySQL memory count with pgvector distinct `source_memory_id` count.
- Do not change `MEMORY_BACKEND` during backfill.
- Do not delete pgvector rows during app rollback without an explicit retention decision.

## Phase 4: pgvector Resolve Canary

Only start canary after schema, backfill, and retrieval quality checks pass.

Environment:

```bash
MEMORY_ENABLED=1
MEMORY_BACKEND=pgvector
MEMORY_VECTOR_ENABLED=1
MEMORY_FAIL_OPEN=1
MEMORY_PGVECTOR_DATABASE_URL='postgresql://...'
MEMORY_PGVECTOR_EMBEDDING_MODULE='./scripts/embed-memory-v2-pgvector.mjs'
```

Health gate:

```bash
npm run check:memory-v2 -- --require-enabled --expect-backend pgvector
```

Required invariant:

- `selectedBackend=pgvector` is allowed for `resolve`.
- `write_uses_legacy` must still pass.
- `compact_uses_legacy` must still pass.
- Any pgvector error must fail open to legacy or degraded empty resolve.

Rollback:

```bash
MEMORY_BACKEND=legacy
MEMORY_VECTOR_ENABLED=0
```

Emergency rollback:

```bash
MEMORY_ENABLED=0
```