memind/docs/memory-v2/production-rollout-runbook.md

Memory V2 Production Rollout Runbook

Memory V2 is a facade and policy layer. Production rollout must not introduce new goosed, PG session, SSE, or Portal execution paths.

Release Gates

Run the normal repository release checks first:

bash scripts/check-release-ready.sh

Run the Memory V2 backend contract gate:

npm run check:memory-v2-contracts

Run the Memory V2 health gate:

npm run check:memory-v2 -- --require-enabled --expect-backend legacy

Expected phase-one result:

• ok=true
• contract gate reports unique backend names and legacy backend present
• summary.enabled=true
• summary.backend=legacy
• summary.selectedBackend=legacy-conversation-memory
• fail_open passes
• legacy_write_compact_supported passes
• write_uses_legacy passes
• compact_uses_legacy passes

When validating the real pgvector adapter before canary:

npm run check:memory-v2-contracts -- --include-pgvector-adapter

Future backend adapter rule:

npm run scaffold:memory-v2-backend -- \
  --name qdrant \
  --category semantic \
  --capability resolve

Use the scaffold as the starting point, add focused adapter tests, then pass npm run check:memory-v2-contracts before wiring any external SDK into runtime. Do not wire external services directly into server.mjs, tkmind-proxy.mjs, goosed, SSE, or Portal execution code.

Qdrant Read-Only Phase

The Qdrant adapter may be visible in runtime status before user traffic is canaried.

Fallback-only environment:

MEMORY_ENABLED=1
MEMORY_BACKEND=qdrant
MEMORY_QDRANT_ENABLED=1
MEMORY_QDRANT_URL='http://qdrant.example:6333'
MEMORY_QDRANT_COLLECTION='memind_memory'

Expected without embedding module:

• memory.backend=qdrant
• memory.selectedBackend=legacy-conversation-memory
• qdrant.available=false
• qdrant.reason=embedding_module_not_configured
• write_uses_legacy still passes
• compact_uses_legacy still passes

Read-only smoke:

MEMORY_QDRANT_URL='http://qdrant.example:6333' \
npm run smoke:memory-v2-qdrant -- \
  --collection memind_memory \
  --vector 1,0,0 \
  --limit 1

The smoke command sends a Qdrant search request only. It must not create collections, write points, or run from app startup.

Runtime read-only canary requires:

MEMORY_QDRANT_EMBEDDING_MODULE='./scripts/embed-memory-v2-qdrant.mjs'

Even during read-only canary, write_uses_legacy and compact_uses_legacy must still pass.

Mem0 Adapter Phase

Mem0 is the Memory Extraction backend boundary. The adapter uses HTTP and is dormant unless explicitly configured.

Environment:

MEMORY_ENABLED=1
MEMORY_BACKEND=mem0
MEMORY_MEM0_ENABLED=1
MEMORY_MEM0_API_KEY='...'
MEMORY_MEM0_PROJECT_ID='memind_project'
MEMORY_MEM0_BASE_URL='https://api.mem0.ai'

Smoke:

npm run smoke:memory-v2-external -- --backend mem0 --operation write

Do not canary Mem0 writes until sampling, dedupe, rollback, and retention rules are accepted.

Letta Lifecycle Adapter Phase

Letta is the Memory Lifecycle Management backend boundary. The adapter uses HTTP and is dormant unless explicitly configured.

Environment:

MEMORY_ENABLED=1
MEMORY_BACKEND=letta
MEMORY_LETTA_ENABLED=1
MEMORY_LETTA_API_KEY='...'
MEMORY_LETTA_PROJECT_ID='memind_project'
MEMORY_LETTA_AGENT_ID='agent_1'
MEMORY_LETTA_BASE_URL='https://api.letta.com'

Smoke:

npm run smoke:memory-v2-external -- --backend letta --operation resolve

Do not canary Letta until ownership boundaries, long/short-term promotion rules, conflict resolution with conversation-memory, rollback, and retention are accepted.

Remaining External Adapter Phase

Weaviate, Neo4j, Redis Streams, and LangGraph are wired with runtime clients and remain dormant unless explicitly configured.

Environment examples:

MEMORY_WEAVIATE_ENABLED=1
MEMORY_WEAVIATE_URL='https://weaviate.example'
MEMORY_WEAVIATE_COLLECTION='MemindMemory'
MEMORY_WEAVIATE_EMBEDDING_MODULE='./scripts/embed-memory-v2-weaviate.mjs'

MEMORY_NEO4J_ENABLED=1
MEMORY_NEO4J_HTTP_URL='http://localhost:7474'
MEMORY_NEO4J_USER='neo4j'
MEMORY_NEO4J_PASSWORD='...'

MEMORY_REDIS_STREAMS_ENABLED=1
MEMORY_REDIS_STREAMS_URL='redis://localhost:6379'

MEMORY_LANGGRAPH_ENABLED=1
MEMORY_LANGGRAPH_URL='https://langgraph.example'
MEMORY_LANGGRAPH_POLICY_ID='memory_policy'

Smoke:

npm run smoke:memory-v2-external -- --backend weaviate --operation resolve
npm run smoke:memory-v2-external -- --backend neo4j --operation write
npm run smoke:memory-v2-external -- --backend redis-streams --operation write
npm run smoke:memory-v2-external -- --backend langgraph --operation resolve

Future canaries must be backend-specific and must define data ownership, dedupe, observability, rollback, and retention before any adapter can own a Memory V2 operation.

Phase 1: Facade Over Legacy

Environment:

MEMORY_ENABLED=1
MEMORY_BACKEND=legacy
MEMORY_FAIL_OPEN=1
MEMORY_VECTOR_ENABLED=0

Runtime verification:

curl -sk https://<host>/api/runtime/status

Confirm:

• memory.enabled=true
• memory.backend=legacy
• memory.selectedBackend=legacy-conversation-memory
• future plugin slots report available=false

Rollback:

MEMORY_ENABLED=0

Phase 2: Empty pgvector Schema

This phase creates schema only. It must not change runtime backend selection.

Preconditions:

• PostgreSQL server is approved for Memory V2 semantic memory.
• pgvector extension is available.
• Backup/rollback decision is recorded by ops.

Dry-run:

node scripts/setup-memory-v2-pgvector-schema.mjs \
  --table memory_embeddings \
  --dimensions 1536 \
  --create-extension

Apply only through the approved production migration path:

MEMORY_PGVECTOR_DATABASE_URL='postgresql://...' \
node scripts/setup-memory-v2-pgvector-schema.mjs \
  --apply \
  --table memory_embeddings \
  --dimensions 1536 \
  --create-extension

Keep runtime on legacy:

MEMORY_BACKEND=legacy
MEMORY_VECTOR_ENABLED=0

Post-migration smoke verification:

MEMORY_PGVECTOR_DATABASE_URL='postgresql://...' \
npm run smoke:memory-v2-pgvector -- \
  --table memory_embeddings \
  --dimensions 1536

The smoke command inserts synthetic memory-v2-smoke-* rows, verifies nearest-vector retrieval, and deletes those rows by default. In production it should run only after the schema migration. Do not pass --create-schema in production unless the approved migration procedure explicitly requires it.

For local-only plumbing checks, use the deterministic hash embedding module:

MEMORY_PGVECTOR_EMBEDDING_MODULE=./scripts/embed-memory-v2-local-hash.mjs
MEMORY_V2_LOCAL_EMBEDDING_DIMENSIONS=3

Do not use this local hash embedding module for production retrieval quality.

Phase 3: Backfill

Backfill must be separate from schema creation and application rollout.

Dry-run:

MEMORY_BACKFILL_MYSQL_URL='mysql://...' \
node scripts/backfill-memory-v2-pgvector.mjs --limit 100

Apply:

MEMORY_BACKFILL_MYSQL_URL='mysql://...' \
MEMORY_PGVECTOR_DATABASE_URL='postgresql://...' \
node scripts/backfill-memory-v2-pgvector.mjs \
  --apply \
  --limit 100 \
  --embedding-module ./scripts/embed-memory-v2-pgvector.mjs

Operational rules:

• Store { updatedAt, id } checkpoint externally after each successful batch.
• Compare active MySQL memory count with pgvector distinct source_memory_id count.
• Do not change MEMORY_BACKEND during backfill.
• Do not delete pgvector rows during app rollback without an explicit retention decision.

Phase 4: pgvector Resolve Canary

Only start canary after schema, backfill, and retrieval quality checks pass.

Environment:

MEMORY_ENABLED=1
MEMORY_BACKEND=pgvector
MEMORY_VECTOR_ENABLED=1
MEMORY_FAIL_OPEN=1
MEMORY_PGVECTOR_DATABASE_URL='postgresql://...'
MEMORY_PGVECTOR_EMBEDDING_MODULE='./scripts/embed-memory-v2-pgvector.mjs'

Health gate:

npm run check:memory-v2 -- --require-enabled --expect-backend pgvector

Required invariant:

• selectedBackend=pgvector is allowed for resolve.
• write_uses_legacy must still pass.
• compact_uses_legacy must still pass.
• Any pgvector error must fail open to legacy or degraded empty resolve.

Rollback:

MEMORY_BACKEND=legacy
MEMORY_VECTOR_ENABLED=0

Emergency rollback:

MEMORY_ENABLED=0