feat(orchestrator): harden zero-impact shadow rollout
Gate and bound Portal shadow observations while preserving Native execution. Add fail-closed service boundaries, terminal retention controls, Canary readiness telemetry, ops visibility, and isolated regression coverage.
This commit is contained in:
@@ -7,6 +7,7 @@ making it the default executor. The default remains:
|
||||
|
||||
```text
|
||||
memindadm executionEnabled=false
|
||||
Portal MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=0
|
||||
Portal MEMIND_ORCHESTRATOR_EXECUTION_HANDOFF_ENABLED=0
|
||||
Orchestrator MEMIND_ORCHESTRATOR_EXECUTION_ENABLED=0
|
||||
enabled executor list empty
|
||||
@@ -50,7 +51,11 @@ enabled and no non-stale worker heartbeat exists.
|
||||
## Rollout order
|
||||
|
||||
1. `off`: verify storage, metrics, backup, and worker registration.
|
||||
2. `shadow`: collect Native versus LangGraph planning evidence.
|
||||
2. Set Portal `MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=1`, restart only
|
||||
the local/target Portal instance, then select `shadow` to collect Native
|
||||
versus LangGraph planning evidence. Confirm memindadm shows `Shadow wiring:
|
||||
已开启`; Canary readiness must include a passing `shadow_wiring_enabled`
|
||||
check before evidence collection is considered valid.
|
||||
3. `canary`, execution switch off: verify selection denominator and readiness.
|
||||
4. `canary`, execution switch on: one explicit user, one workspace alias, zero
|
||||
percentage rollout.
|
||||
@@ -71,10 +76,11 @@ Use any one of these independent controls:
|
||||
|
||||
1. Set `MEMIND_ORCHESTRATOR_KILL_SWITCH=1` on Portal.
|
||||
2. Clear the memindadm execution switch or set mode to `off`.
|
||||
3. Set Portal `MEMIND_ORCHESTRATOR_EXECUTION_HANDOFF_ENABLED=0`.
|
||||
4. Set Orchestrator `MEMIND_ORCHESTRATOR_EXECUTION_ENABLED=0`.
|
||||
5. Remove an executor from `MEMIND_ORCHESTRATOR_ENABLED_EXECUTORS`.
|
||||
6. Drain a worker by stopping it gracefully; the worker records `draining=true`.
|
||||
3. Set Portal `MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=0`.
|
||||
4. Set Portal `MEMIND_ORCHESTRATOR_EXECUTION_HANDOFF_ENABLED=0`.
|
||||
5. Set Orchestrator `MEMIND_ORCHESTRATOR_EXECUTION_ENABLED=0`.
|
||||
6. Remove an executor from `MEMIND_ORCHESTRATOR_ENABLED_EXECUTORS`.
|
||||
7. Drain a worker by stopping it gracefully; the worker records `draining=true`.
|
||||
|
||||
Queued jobs stay durable. Running jobs stop receiving heartbeats, and lease
|
||||
recovery moves them to `retryable` or `timed_out` according to attempt limits.
|
||||
@@ -88,12 +94,52 @@ Scrape `/metrics` and alert on:
|
||||
- `memind_orchestrator_executor_expired_leases > 0`;
|
||||
- claimable jobs increasing while healthy workers are zero;
|
||||
- repeated `retryable`, `failed`, or `timed_out` states;
|
||||
- any `workflow_shadow_skipped` event or non-zero Shadow skip rate;
|
||||
- `shadow_wiring_enabled` becoming false while memindadm mode remains Shadow;
|
||||
- worker heartbeat age beyond 60 seconds;
|
||||
- any execution-enabled interval without durable PostgreSQL readiness.
|
||||
|
||||
Executor events contain bounded metadata and artifact references. They must not
|
||||
contain provider keys, absolute host paths, full stdout/stderr, or binary data.
|
||||
|
||||
## Data lifecycle
|
||||
|
||||
Shadow uses `control-plane-only-v1`: user content, user ID, and session ID are
|
||||
not copied into Orchestrator storage. The product run ID remains the deletion
|
||||
join key.
|
||||
|
||||
Before deleting the corresponding Memind run or completing a user-data purge,
|
||||
call the authenticated endpoint for every linked run:
|
||||
|
||||
```text
|
||||
DELETE /v1/runs/:runId
|
||||
```
|
||||
|
||||
Only terminal runs can be deleted. The endpoint deletes the LangGraph
|
||||
checkpoint thread and linked terminal Executor Job; PostgreSQL cascades its job
|
||||
events. A `409` means execution is still active and must first be cancelled or
|
||||
allowed to reach a terminal state. This endpoint is internal and must never be
|
||||
exposed without the service-token boundary.
|
||||
|
||||
Bulk retention remains off unless explicitly configured:
|
||||
|
||||
```text
|
||||
MEMIND_ORCHESTRATOR_RETENTION_DAYS=30
|
||||
MEMIND_ORCHESTRATOR_RETENTION_SWEEP_INTERVAL_MS=86400000
|
||||
MEMIND_ORCHESTRATOR_RETENTION_SWEEP_LIMIT=100
|
||||
```
|
||||
|
||||
Before enabling it, preview the exact terminal candidates through the
|
||||
authenticated endpoint. Mutation requires `apply=true`; omitting it is always
|
||||
dry-run:
|
||||
|
||||
```text
|
||||
POST /v1/maintenance/purge-terminal-runs
|
||||
{"before": <epoch-ms>, "limit": 100}
|
||||
```
|
||||
|
||||
The periodic sweep never runs when retention days is empty or `0`.
|
||||
|
||||
## Backup and disaster recovery
|
||||
|
||||
Create and verify a PostgreSQL custom-format backup:
|
||||
|
||||
Reference in New Issue
Block a user