john
6df82818c5
Gate and bound Portal shadow observations while preserving Native execution. Add fail-closed service boundaries, terminal retention controls, Canary readiness telemetry, ops visibility, and isolated regression coverage.
178 lines
6.6 KiB
Markdown
178 lines
6.6 KiB
Markdown
# Workflow Orchestrator Phase 5 Runbook
|
|
|
|
## Scope
|
|
|
|
Phase 5 makes the LangGraph control plane operationally deployable without
|
|
making it the default executor. The default remains:
|
|
|
|
```text
|
|
memindadm executionEnabled=false
|
|
Portal MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=0
|
|
Portal MEMIND_ORCHESTRATOR_EXECUTION_HANDOFF_ENABLED=0
|
|
Orchestrator MEMIND_ORCHESTRATOR_EXECUTION_ENABLED=0
|
|
enabled executor list empty
|
|
Native Agent Run owns execution
|
|
```
|
|
|
|
An execution handoff requires all gates to agree: memindadm mode and explicit
|
|
execution switch, Portal environment gate, deterministic user/workflow rollout,
|
|
healthy Orchestrator storage, Orchestrator execution gate, enabled adapter,
|
|
authorization, workspace alias policy, quota admission, and a healthy worker.
|
|
|
|
## Local Colima canary
|
|
|
|
1. Copy `deploy/orchestrator/.env.example` to the ignored local `.env`.
|
|
2. Set distinct PostgreSQL, service, and worker secrets.
|
|
3. Keep execution disabled and start the durable control plane:
|
|
|
|
```bash
|
|
docker compose -f deploy/orchestrator/compose.yaml up -d --build
|
|
curl -fsS http://127.0.0.1:8093/ready
|
|
curl -fsS http://127.0.0.1:8093/metrics
|
|
```
|
|
|
|
4. Configure a workspace alias understood by the target Goosed instance.
|
|
5. Enable only Goosed for a single local user and start the worker profile:
|
|
|
|
```text
|
|
MEMIND_ORCHESTRATOR_EXECUTION_ENABLED=1
|
|
MEMIND_ORCHESTRATOR_ENABLED_EXECUTORS=goosed
|
|
MEMIND_ORCHESTRATOR_USER_ALLOWLIST=<local-user-id>
|
|
MEMIND_EXECUTOR_WORKSPACE_ALIASES_JSON={"canary":"/absolute/canary/path"}
|
|
```
|
|
|
|
```bash
|
|
docker compose -f deploy/orchestrator/compose.yaml --profile goosed-worker up -d
|
|
```
|
|
|
|
The Orchestrator `/ready` endpoint intentionally fails while execution is
|
|
enabled and no non-stale worker heartbeat exists.
|
|
|
|
## Rollout order
|
|
|
|
1. `off`: verify storage, metrics, backup, and worker registration.
|
|
2. Set Portal `MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=1`, restart only
|
|
the local/target Portal instance, then select `shadow` to collect Native
|
|
versus LangGraph planning evidence. Confirm memindadm shows `Shadow wiring:
|
|
已开启`; Canary readiness must include a passing `shadow_wiring_enabled`
|
|
check before evidence collection is considered valid.
|
|
3. `canary`, execution switch off: verify selection denominator and readiness.
|
|
4. `canary`, execution switch on: one explicit user, one workspace alias, zero
|
|
percentage rollout.
|
|
5. Increase percentage only after success rate, timeout, expired lease, queue
|
|
age, fallback, and Native rollback signals remain within the approved SLO.
|
|
6. `active` remains workflow-allowlisted and requires a separate release
|
|
approval.
|
|
|
|
Never enable Aider or OpenHands in the Orchestrator service until its dedicated
|
|
worker is registered and healthy. Use the example worker Compose overlay as a
|
|
template; each tool gets a separate image, resource limits, writable workspace
|
|
root, read-only container root, no Docker socket, dropped capabilities, and a
|
|
distinct worker identity.
|
|
|
|
## Immediate rollback
|
|
|
|
Use any one of these independent controls:
|
|
|
|
1. Set `MEMIND_ORCHESTRATOR_KILL_SWITCH=1` on Portal.
|
|
2. Clear the memindadm execution switch or set mode to `off`.
|
|
3. Set Portal `MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=0`.
|
|
4. Set Portal `MEMIND_ORCHESTRATOR_EXECUTION_HANDOFF_ENABLED=0`.
|
|
5. Set Orchestrator `MEMIND_ORCHESTRATOR_EXECUTION_ENABLED=0`.
|
|
6. Remove an executor from `MEMIND_ORCHESTRATOR_ENABLED_EXECUTORS`.
|
|
7. Drain a worker by stopping it gracefully; the worker records `draining=true`.
|
|
|
|
Queued jobs stay durable. Running jobs stop receiving heartbeats, and lease
|
|
recovery moves them to `retryable` or `timed_out` according to attempt limits.
|
|
Native fallback remains a Portal decision; the Orchestrator never launches a
|
|
second Native run by itself.
|
|
|
|
## Monitoring
|
|
|
|
Scrape `/metrics` and alert on:
|
|
|
|
- `memind_orchestrator_executor_expired_leases > 0`;
|
|
- claimable jobs increasing while healthy workers are zero;
|
|
- repeated `retryable`, `failed`, or `timed_out` states;
|
|
- any `workflow_shadow_skipped` event or non-zero Shadow skip rate;
|
|
- `shadow_wiring_enabled` becoming false while memindadm mode remains Shadow;
|
|
- worker heartbeat age beyond 60 seconds;
|
|
- any execution-enabled interval without durable PostgreSQL readiness.
|
|
|
|
Executor events contain bounded metadata and artifact references. They must not
|
|
contain provider keys, absolute host paths, full stdout/stderr, or binary data.
|
|
|
|
## Data lifecycle
|
|
|
|
Shadow uses `control-plane-only-v1`: user content, user ID, and session ID are
|
|
not copied into Orchestrator storage. The product run ID remains the deletion
|
|
join key.
|
|
|
|
Before deleting the corresponding Memind run or completing a user-data purge,
|
|
call the authenticated endpoint for every linked run:
|
|
|
|
```text
|
|
DELETE /v1/runs/:runId
|
|
```
|
|
|
|
Only terminal runs can be deleted. The endpoint deletes the LangGraph
|
|
checkpoint thread and linked terminal Executor Job; PostgreSQL cascades its job
|
|
events. A `409` means execution is still active and must first be cancelled or
|
|
allowed to reach a terminal state. This endpoint is internal and must never be
|
|
exposed without the service-token boundary.
|
|
|
|
Bulk retention remains off unless explicitly configured:
|
|
|
|
```text
|
|
MEMIND_ORCHESTRATOR_RETENTION_DAYS=30
|
|
MEMIND_ORCHESTRATOR_RETENTION_SWEEP_INTERVAL_MS=86400000
|
|
MEMIND_ORCHESTRATOR_RETENTION_SWEEP_LIMIT=100
|
|
```
|
|
|
|
Before enabling it, preview the exact terminal candidates through the
|
|
authenticated endpoint. Mutation requires `apply=true`; omitting it is always
|
|
dry-run:
|
|
|
|
```text
|
|
POST /v1/maintenance/purge-terminal-runs
|
|
{"before": <epoch-ms>, "limit": 100}
|
|
```
|
|
|
|
The periodic sweep never runs when retention days is empty or `0`.
|
|
|
|
## Backup and disaster recovery
|
|
|
|
Create and verify a PostgreSQL custom-format backup:
|
|
|
|
```bash
|
|
node scripts/orchestrator-dr.mjs backup --output /safe/path/orchestrator.dump
|
|
node scripts/orchestrator-dr.mjs verify --input /safe/path/orchestrator.dump
|
|
```
|
|
|
|
Restore only into a confirmed empty target:
|
|
|
|
```bash
|
|
node scripts/orchestrator-dr.mjs restore \
|
|
--input /safe/path/orchestrator.dump \
|
|
--database-url postgresql://... \
|
|
--confirm-empty-target
|
|
```
|
|
|
|
The restore command never uses `--clean` and refuses remote targets unless
|
|
`--allow-remote-target` is explicitly supplied. After restore, keep execution
|
|
disabled, start one worker, verify checkpoints/jobs/events/worker heartbeats,
|
|
then repeat the rollout order from `off`.
|
|
|
|
## Production release gate
|
|
|
|
This runbook does not authorize a production action. Before any server-side
|
|
deployment, follow `ENGINEERING_WORKFLOW_RULES.md`,
|
|
`PRODUCTION_RELEASE_RULES.md`, and run:
|
|
|
|
```bash
|
|
bash scripts/check-release-ready.sh
|
|
```
|
|
|
|
Production still requires a clean commit on complete `main`, passing CI, an
|
|
explicit deployment approval, a current backup, and a tested rollback.
|