memind/docs/orchestrator-phase5-runbook.md

Workflow Orchestrator Phase 5 Runbook

Scope

Phase 5 makes the LangGraph control plane operationally deployable without making it the default executor. The default remains:

memindadm executionEnabled=false
Portal MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=0
Portal MEMIND_ORCHESTRATOR_EXECUTION_HANDOFF_ENABLED=0
Orchestrator MEMIND_ORCHESTRATOR_EXECUTION_ENABLED=0
enabled executor list empty
Native Agent Run owns execution

An execution handoff requires all gates to agree: memindadm mode and explicit execution switch, Portal environment gate, deterministic user/workflow rollout, healthy Orchestrator storage, Orchestrator execution gate, enabled adapter, authorization, workspace alias policy, quota admission, and a healthy worker.

Local Colima canary

1. Copy deploy/orchestrator/.env.example to the ignored local .env.
2. Set distinct PostgreSQL, service, and worker secrets.
3. Keep execution disabled and start the durable control plane:

docker compose -f deploy/orchestrator/compose.yaml up -d --build
curl -fsS http://127.0.0.1:8093/ready
curl -fsS http://127.0.0.1:8093/metrics

4. Configure a workspace alias understood by the target Goosed instance.
5. Enable only Goosed for a single local user and start the worker profile:

MEMIND_ORCHESTRATOR_EXECUTION_ENABLED=1
MEMIND_ORCHESTRATOR_ENABLED_EXECUTORS=goosed
MEMIND_ORCHESTRATOR_USER_ALLOWLIST=<local-user-id>
MEMIND_EXECUTOR_WORKSPACE_ALIASES_JSON={"canary":"/absolute/canary/path"}

docker compose -f deploy/orchestrator/compose.yaml --profile goosed-worker up -d

The Orchestrator /ready endpoint intentionally fails while execution is enabled and no non-stale worker heartbeat exists.

Rollout order

1. off: verify storage, metrics, backup, and worker registration.
2. Set Portal MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=1, restart only the local/target Portal instance, then select shadow to collect Native versus LangGraph planning evidence. Confirm memindadm shows Shadow wiring: 已开启; Canary readiness must include a passing shadow_wiring_enabled check before evidence collection is considered valid.
3. canary, execution switch off: verify selection denominator and readiness.
4. canary, execution switch on: one explicit user, one workspace alias, zero percentage rollout.
5. Increase percentage only after success rate, timeout, expired lease, queue age, fallback, and Native rollback signals remain within the approved SLO.
6. active remains workflow-allowlisted and requires a separate release approval.

Never enable Aider or OpenHands in the Orchestrator service until its dedicated worker is registered and healthy. Use the example worker Compose overlay as a template; each tool gets a separate image, resource limits, writable workspace root, read-only container root, no Docker socket, dropped capabilities, and a distinct worker identity.

Immediate rollback

Use any one of these independent controls:

1. Set MEMIND_ORCHESTRATOR_KILL_SWITCH=1 on Portal.
2. Clear the memindadm execution switch or set mode to off.
3. Set Portal MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=0.
4. Set Portal MEMIND_ORCHESTRATOR_EXECUTION_HANDOFF_ENABLED=0.
5. Set Orchestrator MEMIND_ORCHESTRATOR_EXECUTION_ENABLED=0.
6. Remove an executor from MEMIND_ORCHESTRATOR_ENABLED_EXECUTORS.
7. Drain a worker by stopping it gracefully; the worker records draining=true.

Queued jobs stay durable. Running jobs stop receiving heartbeats, and lease recovery moves them to retryable or timed_out according to attempt limits. Native fallback remains a Portal decision; the Orchestrator never launches a second Native run by itself.

Monitoring

Scrape /metrics and alert on:

• memind_orchestrator_executor_expired_leases > 0;
• claimable jobs increasing while healthy workers are zero;
• repeated retryable, failed, or timed_out states;
• any workflow_shadow_skipped event or non-zero Shadow skip rate;
• shadow_wiring_enabled becoming false while memindadm mode remains Shadow;
• worker heartbeat age beyond 60 seconds;
• any execution-enabled interval without durable PostgreSQL readiness.

Executor events contain bounded metadata and artifact references. They must not contain provider keys, absolute host paths, full stdout/stderr, or binary data.

Data lifecycle

Shadow uses control-plane-only-v1: user content, user ID, and session ID are not copied into Orchestrator storage. The product run ID remains the deletion join key.

Before deleting the corresponding Memind run or completing a user-data purge, call the authenticated endpoint for every linked run:

DELETE /v1/runs/:runId

Only terminal runs can be deleted. The endpoint deletes the LangGraph checkpoint thread and linked terminal Executor Job; PostgreSQL cascades its job events. A 409 means execution is still active and must first be cancelled or allowed to reach a terminal state. This endpoint is internal and must never be exposed without the service-token boundary.

Bulk retention remains off unless explicitly configured:

MEMIND_ORCHESTRATOR_RETENTION_DAYS=30
MEMIND_ORCHESTRATOR_RETENTION_SWEEP_INTERVAL_MS=86400000
MEMIND_ORCHESTRATOR_RETENTION_SWEEP_LIMIT=100

Before enabling it, preview the exact terminal candidates through the authenticated endpoint. Mutation requires apply=true; omitting it is always dry-run:

POST /v1/maintenance/purge-terminal-runs
{"before": <epoch-ms>, "limit": 100}

The periodic sweep never runs when retention days is empty or 0.

Backup and disaster recovery

Create and verify a PostgreSQL custom-format backup:

node scripts/orchestrator-dr.mjs backup --output /safe/path/orchestrator.dump
node scripts/orchestrator-dr.mjs verify --input /safe/path/orchestrator.dump

Restore only into a confirmed empty target:

node scripts/orchestrator-dr.mjs restore \
  --input /safe/path/orchestrator.dump \
  --database-url postgresql://... \
  --confirm-empty-target

The restore command never uses --clean and refuses remote targets unless --allow-remote-target is explicitly supplied. After restore, keep execution disabled, start one worker, verify checkpoints/jobs/events/worker heartbeats, then repeat the rollout order from off.

Production release gate

This runbook does not authorize a production action. Before any server-side deployment, follow ENGINEERING_WORKFLOW_RULES.md, PRODUCTION_RELEASE_RULES.md, and run:

bash scripts/check-release-ready.sh

Production still requires a clean commit on complete main, passing CI, an explicit deployment approval, a current backup, and a tested rollback.