feat(orchestrator): harden zero-impact shadow rollout

Gate and bound Portal shadow observations while preserving Native execution. Add fail-closed service boundaries, terminal retention controls, Canary readiness telemetry, ops visibility, and isolated regression coverage.
This commit is contained in:
john
2026-07-25 07:28:37 +08:00
parent 08a48e4849
commit 6df82818c5
33 changed files with 1569 additions and 108 deletions
@@ -54,6 +54,18 @@ Native selection.
The initial workflow allowlist contains only `code-run-v1`. Ordinary chat and
existing Goosed session traffic remain outside the Orchestrator path.
Portal also has an independent wiring gate:
`MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=1`. It is off by default.
When it is off, Portal does not construct the Shadow observer, query routing
configuration from the Agent Run path, schedule background observations, or
send data to the Orchestrator. Changing memindadm mode alone therefore cannot
put existing Native traffic on the Shadow path.
memindadm projects the requested and effective wiring states separately. Canary
readiness requires `shadow_wiring_enabled=true`, so a saved Shadow mode with a
closed Portal gate is visible and cannot be mistaken for a collecting Shadow
deployment.
Phase 2 implemented only the `off` and `shadow` execution semantics. Canary and
Active were initially planning-only so an administrative configuration mistake
could not create two task executors.
@@ -145,6 +157,8 @@ POST /v1/runs
GET /v1/runs/:id
POST /v1/runs/:id/resume
POST /v1/runs/:id/cancel
DELETE /v1/runs/:id
POST /v1/maintenance/purge-terminal-runs
GET /v1/runs/:id/events?after=<cursor>
GET /v1/executor-jobs/:jobId
GET /v1/executor-jobs/:jobId/events?after=<cursor>&limit=<limit>
@@ -154,6 +168,15 @@ External state changes must use an idempotency key derived from run, graph
version, node, and attempt. Graph state stores resource references and decisions,
not API keys, large logs, binary artifacts, or absolute production paths.
Shadow uses the stricter `control-plane-only-v1` data policy. Portal evaluates
the rollout locally but sends no user message, user identifier, session
identifier, provider key, or workspace path to LangGraph. The remote RunSpec
contains only product run/request identifiers, workflow/task type, configuration
version, and a fixed non-user instruction marker. Terminal runs can be removed
through `DELETE /v1/runs/:id`; deletion removes the LangGraph thread and its
linked terminal Executor Job, whose events cascade in PostgreSQL. Active or
waiting runs fail deletion with `409`.
The graph keeps three deterministic control-plane nodes:
```text
@@ -205,10 +228,29 @@ failure:
4. remains removable by deleting the observer wiring and changing only the
service URL boundary.
The observer uses a bounded in-process dispatcher. Concurrency defaults to `2`
and the waiting queue to `100`; overflow records `workflow_shadow_skipped` and
never blocks, rejects, or cancels the Native run. Skipped observations are
included in the denominator, shown separately in memindadm, and any non-zero
skip rate blocks Canary readiness.
Successful observations are projected as `workflow_shadow_completed`; graph
checkpoints and blocked Executor Jobs stay in the Orchestrator-owned PostgreSQL
database.
Non-loopback Orchestrator binding requires a service token. Enabling Executor
Jobs requires both distinct service and worker tokens even on loopback.
Non-loopback service URLs are rejected by Portal unless their origin is
declared through `MEMIND_ORCHESTRATOR_URL` or
`MEMIND_ORCHESTRATOR_ALLOWED_ORIGINS`; loopback origins remain allowed for
local validation.
Terminal retention is independently disabled by default. Setting
`MEMIND_ORCHESTRATOR_RETENTION_DAYS` to a positive value starts a bounded
periodic sweep after service startup; it uses the same terminal-only deletion
path and never deletes waiting/running workflows. The authenticated maintenance
endpoint defaults to dry-run and requires `apply=true` for mutation.
Both terminal projection events contain bounded observation latency. The
Memind-owned observability service aggregates those events and may join them to
the product run status. Only per-run detail calls cross the service boundary to
@@ -218,6 +260,7 @@ Canary readiness is also a Memind control-plane projection. It excludes
synthetic smoke runs and evaluates operational health, durable checkpoint state,
durable Executor Job storage, sample volume, session coverage, success rate,
latency coverage, P95 latency, Native terminal coverage, and sample freshness.
It also requires the independent Portal Shadow wiring gate to be effective.
The result is advisory:
`manual_canary_review` never mutates routing configuration or transfers
execution ownership.