feat(orchestrator): harden zero-impact shadow rollout

Gate and bound Portal shadow observations while preserving Native execution. Add fail-closed service boundaries, terminal retention controls, Canary readiness telemetry, ops visibility, and isolated regression coverage.
This commit is contained in:
john
2026-07-25 07:28:37 +08:00
parent 08a48e4849
commit 6df82818c5
33 changed files with 1569 additions and 108 deletions
+51 -5
View File
@@ -7,6 +7,7 @@ making it the default executor. The default remains:
```text
memindadm executionEnabled=false
Portal MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=0
Portal MEMIND_ORCHESTRATOR_EXECUTION_HANDOFF_ENABLED=0
Orchestrator MEMIND_ORCHESTRATOR_EXECUTION_ENABLED=0
enabled executor list empty
@@ -50,7 +51,11 @@ enabled and no non-stale worker heartbeat exists.
## Rollout order
1. `off`: verify storage, metrics, backup, and worker registration.
2. `shadow`: collect Native versus LangGraph planning evidence.
2. Set Portal `MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=1`, restart only
the local/target Portal instance, then select `shadow` to collect Native
versus LangGraph planning evidence. Confirm memindadm shows `Shadow wiring:
已开启`; Canary readiness must include a passing `shadow_wiring_enabled`
check before evidence collection is considered valid.
3. `canary`, execution switch off: verify selection denominator and readiness.
4. `canary`, execution switch on: one explicit user, one workspace alias, zero
percentage rollout.
@@ -71,10 +76,11 @@ Use any one of these independent controls:
1. Set `MEMIND_ORCHESTRATOR_KILL_SWITCH=1` on Portal.
2. Clear the memindadm execution switch or set mode to `off`.
3. Set Portal `MEMIND_ORCHESTRATOR_EXECUTION_HANDOFF_ENABLED=0`.
4. Set Orchestrator `MEMIND_ORCHESTRATOR_EXECUTION_ENABLED=0`.
5. Remove an executor from `MEMIND_ORCHESTRATOR_ENABLED_EXECUTORS`.
6. Drain a worker by stopping it gracefully; the worker records `draining=true`.
3. Set Portal `MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=0`.
4. Set Portal `MEMIND_ORCHESTRATOR_EXECUTION_HANDOFF_ENABLED=0`.
5. Set Orchestrator `MEMIND_ORCHESTRATOR_EXECUTION_ENABLED=0`.
6. Remove an executor from `MEMIND_ORCHESTRATOR_ENABLED_EXECUTORS`.
7. Drain a worker by stopping it gracefully; the worker records `draining=true`.
Queued jobs stay durable. Running jobs stop receiving heartbeats, and lease
recovery moves them to `retryable` or `timed_out` according to attempt limits.
@@ -88,12 +94,52 @@ Scrape `/metrics` and alert on:
- `memind_orchestrator_executor_expired_leases > 0`;
- claimable jobs increasing while healthy workers are zero;
- repeated `retryable`, `failed`, or `timed_out` states;
- any `workflow_shadow_skipped` event or non-zero Shadow skip rate;
- `shadow_wiring_enabled` becoming false while memindadm mode remains Shadow;
- worker heartbeat age beyond 60 seconds;
- any execution-enabled interval without durable PostgreSQL readiness.
Executor events contain bounded metadata and artifact references. They must not
contain provider keys, absolute host paths, full stdout/stderr, or binary data.
## Data lifecycle
Shadow uses `control-plane-only-v1`: user content, user ID, and session ID are
not copied into Orchestrator storage. The product run ID remains the deletion
join key.
Before deleting the corresponding Memind run or completing a user-data purge,
call the authenticated endpoint for every linked run:
```text
DELETE /v1/runs/:runId
```
Only terminal runs can be deleted. The endpoint deletes the LangGraph
checkpoint thread and linked terminal Executor Job; PostgreSQL cascades its job
events. A `409` means execution is still active and must first be cancelled or
allowed to reach a terminal state. This endpoint is internal and must never be
exposed without the service-token boundary.
Bulk retention remains off unless explicitly configured:
```text
MEMIND_ORCHESTRATOR_RETENTION_DAYS=30
MEMIND_ORCHESTRATOR_RETENTION_SWEEP_INTERVAL_MS=86400000
MEMIND_ORCHESTRATOR_RETENTION_SWEEP_LIMIT=100
```
Before enabling it, preview the exact terminal candidates through the
authenticated endpoint. Mutation requires `apply=true`; omitting it is always
dry-run:
```text
POST /v1/maintenance/purge-terminal-runs
{"before": <epoch-ms>, "limit": 100}
```
The periodic sweep never runs when retention days is empty or `0`.
## Backup and disaster recovery
Create and verify a PostgreSQL custom-format backup: