feat(orchestrator): harden zero-impact shadow rollout
Gate and bound Portal shadow observations while preserving Native execution. Add fail-closed service boundaries, terminal retention controls, Canary readiness telemetry, ops visibility, and isolated regression coverage.
This commit is contained in:
@@ -54,6 +54,18 @@ Native selection.
|
||||
The initial workflow allowlist contains only `code-run-v1`. Ordinary chat and
|
||||
existing Goosed session traffic remain outside the Orchestrator path.
|
||||
|
||||
Portal also has an independent wiring gate:
|
||||
`MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=1`. It is off by default.
|
||||
When it is off, Portal does not construct the Shadow observer, query routing
|
||||
configuration from the Agent Run path, schedule background observations, or
|
||||
send data to the Orchestrator. Changing memindadm mode alone therefore cannot
|
||||
put existing Native traffic on the Shadow path.
|
||||
|
||||
memindadm projects the requested and effective wiring states separately. Canary
|
||||
readiness requires `shadow_wiring_enabled=true`, so a saved Shadow mode with a
|
||||
closed Portal gate is visible and cannot be mistaken for a collecting Shadow
|
||||
deployment.
|
||||
|
||||
Phase 2 implemented only the `off` and `shadow` execution semantics. Canary and
|
||||
Active were initially planning-only so an administrative configuration mistake
|
||||
could not create two task executors.
|
||||
@@ -145,6 +157,8 @@ POST /v1/runs
|
||||
GET /v1/runs/:id
|
||||
POST /v1/runs/:id/resume
|
||||
POST /v1/runs/:id/cancel
|
||||
DELETE /v1/runs/:id
|
||||
POST /v1/maintenance/purge-terminal-runs
|
||||
GET /v1/runs/:id/events?after=<cursor>
|
||||
GET /v1/executor-jobs/:jobId
|
||||
GET /v1/executor-jobs/:jobId/events?after=<cursor>&limit=<limit>
|
||||
@@ -154,6 +168,15 @@ External state changes must use an idempotency key derived from run, graph
|
||||
version, node, and attempt. Graph state stores resource references and decisions,
|
||||
not API keys, large logs, binary artifacts, or absolute production paths.
|
||||
|
||||
Shadow uses the stricter `control-plane-only-v1` data policy. Portal evaluates
|
||||
the rollout locally but sends no user message, user identifier, session
|
||||
identifier, provider key, or workspace path to LangGraph. The remote RunSpec
|
||||
contains only product run/request identifiers, workflow/task type, configuration
|
||||
version, and a fixed non-user instruction marker. Terminal runs can be removed
|
||||
through `DELETE /v1/runs/:id`; deletion removes the LangGraph thread and its
|
||||
linked terminal Executor Job, whose events cascade in PostgreSQL. Active or
|
||||
waiting runs fail deletion with `409`.
|
||||
|
||||
The graph keeps three deterministic control-plane nodes:
|
||||
|
||||
```text
|
||||
@@ -205,10 +228,29 @@ failure:
|
||||
4. remains removable by deleting the observer wiring and changing only the
|
||||
service URL boundary.
|
||||
|
||||
The observer uses a bounded in-process dispatcher. Concurrency defaults to `2`
|
||||
and the waiting queue to `100`; overflow records `workflow_shadow_skipped` and
|
||||
never blocks, rejects, or cancels the Native run. Skipped observations are
|
||||
included in the denominator, shown separately in memindadm, and any non-zero
|
||||
skip rate blocks Canary readiness.
|
||||
|
||||
Successful observations are projected as `workflow_shadow_completed`; graph
|
||||
checkpoints and blocked Executor Jobs stay in the Orchestrator-owned PostgreSQL
|
||||
database.
|
||||
|
||||
Non-loopback Orchestrator binding requires a service token. Enabling Executor
|
||||
Jobs requires both distinct service and worker tokens even on loopback.
|
||||
Non-loopback service URLs are rejected by Portal unless their origin is
|
||||
declared through `MEMIND_ORCHESTRATOR_URL` or
|
||||
`MEMIND_ORCHESTRATOR_ALLOWED_ORIGINS`; loopback origins remain allowed for
|
||||
local validation.
|
||||
|
||||
Terminal retention is independently disabled by default. Setting
|
||||
`MEMIND_ORCHESTRATOR_RETENTION_DAYS` to a positive value starts a bounded
|
||||
periodic sweep after service startup; it uses the same terminal-only deletion
|
||||
path and never deletes waiting/running workflows. The authenticated maintenance
|
||||
endpoint defaults to dry-run and requires `apply=true` for mutation.
|
||||
|
||||
Both terminal projection events contain bounded observation latency. The
|
||||
Memind-owned observability service aggregates those events and may join them to
|
||||
the product run status. Only per-run detail calls cross the service boundary to
|
||||
@@ -218,6 +260,7 @@ Canary readiness is also a Memind control-plane projection. It excludes
|
||||
synthetic smoke runs and evaluates operational health, durable checkpoint state,
|
||||
durable Executor Job storage, sample volume, session coverage, success rate,
|
||||
latency coverage, P95 latency, Native terminal coverage, and sample freshness.
|
||||
It also requires the independent Portal Shadow wiring gate to be effective.
|
||||
The result is advisory:
|
||||
`manual_canary_review` never mutates routing configuration or transfers
|
||||
execution ownership.
|
||||
|
||||
Reference in New Issue
Block a user