feat(orchestrator): harden zero-impact shadow rollout
Gate and bound Portal shadow observations while preserving Native execution. Add fail-closed service boundaries, terminal retention controls, Canary readiness telemetry, ops visibility, and isolated regression coverage.
This commit is contained in:
john
@@ -54,6 +54,18 @@ Native selection.
|
||||
The initial workflow allowlist contains only `code-run-v1`. Ordinary chat and
|
||||
existing Goosed session traffic remain outside the Orchestrator path.
|
||||
|
||||
Portal also has an independent wiring gate:
|
||||
`MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=1`. It is off by default.
|
||||
When it is off, Portal does not construct the Shadow observer, query routing
|
||||
configuration from the Agent Run path, schedule background observations, or
|
||||
send data to the Orchestrator. Changing memindadm mode alone therefore cannot
|
||||
put existing Native traffic on the Shadow path.
|
||||
|
||||
memindadm projects the requested and effective wiring states separately. Canary
|
||||
readiness requires `shadow_wiring_enabled=true`, so a saved Shadow mode with a
|
||||
closed Portal gate is visible and cannot be mistaken for a collecting Shadow
|
||||
deployment.
|
||||
|
||||
Phase 2 implemented only the `off` and `shadow` execution semantics. Canary and
|
||||
Active were initially planning-only so an administrative configuration mistake
|
||||
could not create two task executors.
|
||||
@@ -145,6 +157,8 @@ POST /v1/runs
|
||||
GET /v1/runs/:id
|
||||
POST /v1/runs/:id/resume
|
||||
POST /v1/runs/:id/cancel
|
||||
DELETE /v1/runs/:id
|
||||
POST /v1/maintenance/purge-terminal-runs
|
||||
GET /v1/runs/:id/events?after=<cursor>
|
||||
GET /v1/executor-jobs/:jobId
|
||||
GET /v1/executor-jobs/:jobId/events?after=<cursor>&limit=<limit>
|
||||
@@ -154,6 +168,15 @@ External state changes must use an idempotency key derived from run, graph
|
||||
version, node, and attempt. Graph state stores resource references and decisions,
|
||||
not API keys, large logs, binary artifacts, or absolute production paths.
|
||||
|
||||
Shadow uses the stricter `control-plane-only-v1` data policy. Portal evaluates
|
||||
the rollout locally but sends no user message, user identifier, session
|
||||
identifier, provider key, or workspace path to LangGraph. The remote RunSpec
|
||||
contains only product run/request identifiers, workflow/task type, configuration
|
||||
version, and a fixed non-user instruction marker. Terminal runs can be removed
|
||||
through `DELETE /v1/runs/:id`; deletion removes the LangGraph thread and its
|
||||
linked terminal Executor Job, whose events cascade in PostgreSQL. Active or
|
||||
waiting runs fail deletion with `409`.
|
||||
|
||||
The graph keeps three deterministic control-plane nodes:
|
||||
|
||||
```text
|
||||
@@ -205,10 +228,29 @@ failure:
|
||||
4. remains removable by deleting the observer wiring and changing only the
|
||||
service URL boundary.
|
||||
|
||||
The observer uses a bounded in-process dispatcher. Concurrency defaults to `2`
|
||||
and the waiting queue to `100`; overflow records `workflow_shadow_skipped` and
|
||||
never blocks, rejects, or cancels the Native run. Skipped observations are
|
||||
included in the denominator, shown separately in memindadm, and any non-zero
|
||||
skip rate blocks Canary readiness.
|
||||
|
||||
Successful observations are projected as `workflow_shadow_completed`; graph
|
||||
checkpoints and blocked Executor Jobs stay in the Orchestrator-owned PostgreSQL
|
||||
database.
|
||||
|
||||
Non-loopback Orchestrator binding requires a service token. Enabling Executor
|
||||
Jobs requires both distinct service and worker tokens even on loopback.
|
||||
Non-loopback service URLs are rejected by Portal unless their origin is
|
||||
declared through `MEMIND_ORCHESTRATOR_URL` or
|
||||
`MEMIND_ORCHESTRATOR_ALLOWED_ORIGINS`; loopback origins remain allowed for
|
||||
local validation.
|
||||
|
||||
Terminal retention is independently disabled by default. Setting
|
||||
`MEMIND_ORCHESTRATOR_RETENTION_DAYS` to a positive value starts a bounded
|
||||
periodic sweep after service startup; it uses the same terminal-only deletion
|
||||
path and never deletes waiting/running workflows. The authenticated maintenance
|
||||
endpoint defaults to dry-run and requires `apply=true` for mutation.
|
||||
|
||||
Both terminal projection events contain bounded observation latency. The
|
||||
Memind-owned observability service aggregates those events and may join them to
|
||||
the product run status. Only per-run detail calls cross the service boundary to
|
||||
@@ -218,6 +260,7 @@ Canary readiness is also a Memind control-plane projection. It excludes
|
||||
synthetic smoke runs and evaluates operational health, durable checkpoint state,
|
||||
durable Executor Job storage, sample volume, session coverage, success rate,
|
||||
latency coverage, P95 latency, Native terminal coverage, and sample freshness.
|
||||
It also requires the independent Portal Shadow wiring gate to be effective.
|
||||
The result is advisory:
|
||||
`manual_canary_review` never mutates routing configuration or transfers
|
||||
execution ownership.
|
||||
|
||||
@@ -7,6 +7,7 @@ making it the default executor. The default remains:
|
||||
|
||||
```text
|
||||
memindadm executionEnabled=false
|
||||
Portal MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=0
|
||||
Portal MEMIND_ORCHESTRATOR_EXECUTION_HANDOFF_ENABLED=0
|
||||
Orchestrator MEMIND_ORCHESTRATOR_EXECUTION_ENABLED=0
|
||||
enabled executor list empty
|
||||
@@ -50,7 +51,11 @@ enabled and no non-stale worker heartbeat exists.
|
||||
## Rollout order
|
||||
|
||||
1. `off`: verify storage, metrics, backup, and worker registration.
|
||||
2. `shadow`: collect Native versus LangGraph planning evidence.
|
||||
2. Set Portal `MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=1`, restart only
|
||||
the local/target Portal instance, then select `shadow` to collect Native
|
||||
versus LangGraph planning evidence. Confirm memindadm shows `Shadow wiring:
|
||||
已开启`; Canary readiness must include a passing `shadow_wiring_enabled`
|
||||
check before evidence collection is considered valid.
|
||||
3. `canary`, execution switch off: verify selection denominator and readiness.
|
||||
4. `canary`, execution switch on: one explicit user, one workspace alias, zero
|
||||
percentage rollout.
|
||||
@@ -71,10 +76,11 @@ Use any one of these independent controls:
|
||||
|
||||
1. Set `MEMIND_ORCHESTRATOR_KILL_SWITCH=1` on Portal.
|
||||
2. Clear the memindadm execution switch or set mode to `off`.
|
||||
3. Set Portal `MEMIND_ORCHESTRATOR_EXECUTION_HANDOFF_ENABLED=0`.
|
||||
4. Set Orchestrator `MEMIND_ORCHESTRATOR_EXECUTION_ENABLED=0`.
|
||||
5. Remove an executor from `MEMIND_ORCHESTRATOR_ENABLED_EXECUTORS`.
|
||||
6. Drain a worker by stopping it gracefully; the worker records `draining=true`.
|
||||
3. Set Portal `MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=0`.
|
||||
4. Set Portal `MEMIND_ORCHESTRATOR_EXECUTION_HANDOFF_ENABLED=0`.
|
||||
5. Set Orchestrator `MEMIND_ORCHESTRATOR_EXECUTION_ENABLED=0`.
|
||||
6. Remove an executor from `MEMIND_ORCHESTRATOR_ENABLED_EXECUTORS`.
|
||||
7. Drain a worker by stopping it gracefully; the worker records `draining=true`.
|
||||
|
||||
Queued jobs stay durable. Running jobs stop receiving heartbeats, and lease
|
||||
recovery moves them to `retryable` or `timed_out` according to attempt limits.
|
||||
@@ -88,12 +94,52 @@ Scrape `/metrics` and alert on:
|
||||
- `memind_orchestrator_executor_expired_leases > 0`;
|
||||
- claimable jobs increasing while healthy workers are zero;
|
||||
- repeated `retryable`, `failed`, or `timed_out` states;
|
||||
- any `workflow_shadow_skipped` event or non-zero Shadow skip rate;
|
||||
- `shadow_wiring_enabled` becoming false while memindadm mode remains Shadow;
|
||||
- worker heartbeat age beyond 60 seconds;
|
||||
- any execution-enabled interval without durable PostgreSQL readiness.
|
||||
|
||||
Executor events contain bounded metadata and artifact references. They must not
|
||||
contain provider keys, absolute host paths, full stdout/stderr, or binary data.
|
||||
|
||||
## Data lifecycle
|
||||
|
||||
Shadow uses `control-plane-only-v1`: user content, user ID, and session ID are
|
||||
not copied into Orchestrator storage. The product run ID remains the deletion
|
||||
join key.
|
||||
|
||||
Before deleting the corresponding Memind run or completing a user-data purge,
|
||||
call the authenticated endpoint for every linked run:
|
||||
|
||||
```text
|
||||
DELETE /v1/runs/:runId
|
||||
```
|
||||
|
||||
Only terminal runs can be deleted. The endpoint deletes the LangGraph
|
||||
checkpoint thread and linked terminal Executor Job; PostgreSQL cascades its job
|
||||
events. A `409` means execution is still active and must first be cancelled or
|
||||
allowed to reach a terminal state. This endpoint is internal and must never be
|
||||
exposed without the service-token boundary.
|
||||
|
||||
Bulk retention remains off unless explicitly configured:
|
||||
|
||||
```text
|
||||
MEMIND_ORCHESTRATOR_RETENTION_DAYS=30
|
||||
MEMIND_ORCHESTRATOR_RETENTION_SWEEP_INTERVAL_MS=86400000
|
||||
MEMIND_ORCHESTRATOR_RETENTION_SWEEP_LIMIT=100
|
||||
```
|
||||
|
||||
Before enabling it, preview the exact terminal candidates through the
|
||||
authenticated endpoint. Mutation requires `apply=true`; omitting it is always
|
||||
dry-run:
|
||||
|
||||
```text
|
||||
POST /v1/maintenance/purge-terminal-runs
|
||||
{"before": <epoch-ms>, "limit": 100}
|
||||
```
|
||||
|
||||
The periodic sweep never runs when retention days is empty or `0`.
|
||||
|
||||
## Backup and disaster recovery
|
||||
|
||||
Create and verify a PostgreSQL custom-format backup:
|
||||
|
||||
Reference in New Issue
Block a user