tkmind/memind
1
0
Fork 0
Code Issues Pull Requests 2 Actions Packages Projects Releases Wiki Activity

feat(orchestrator): harden zero-impact shadow rollout

Browse Source 
Gate and bound Portal shadow observations while preserving Native execution. Add fail-closed service boundaries, terminal retention controls, Canary readiness telemetry, ops visibility, and isolated regression coverage.
This commit is contained in:
john
2026-07-25 07:28:37 +08:00
parent 08a48e4849
commit 6df82818c5
33 changed files with 1569 additions and 108 deletions
Download Patch File Download Diff File Expand all files Collapse all files
services/orchestrator/README.md
+32 -1
View File
@@ -197,7 +197,15 @@ MEMIND_ORCHESTRATOR_CHECKPOINT_MODE=memory pnpm dev:orchestrator
```
The service binds to `127.0.0.1:8093` by default. Configure the same URL and
service token in Portal, then enable `shadow` in `/ops/admin/orchestrator`.
service token in Portal, set
`MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=1` on that Portal instance,
then enable `shadow` in `/ops/admin/orchestrator`. Without the Portal gate,
Native code runs do not construct or call the observer.
The admin runtime projection distinguishes requested Shadow mode from effective
Portal wiring. Canary readiness requires `shadow_wiring_enabled=true`; a closed
environment gate therefore blocks promotion even when older successful samples
are still present.
memindadm exposes the projection through:
@@ -223,8 +231,14 @@ POST /v1/workers/jobs/:jobId/complete
POST /v1/workers/jobs/:jobId/fail
POST /v1/workers/recover-expired
GET /v1/workers/stats
DELETE /v1/runs/:runId
POST /v1/maintenance/purge-terminal-runs
```
Shadow RunSpecs use the `control-plane-only-v1` policy and do not copy user
messages, user IDs, or session IDs into Orchestrator storage. Deleting a
terminal run removes its checkpoint thread and linked terminal Executor Job.
## Colima deployment
Colima is the recommended first container host on macOS because this service and
@@ -278,6 +292,23 @@ requires the repository release gates and a separately approved deployment.
| `MEMIND_ORCHESTRATOR_CHECKPOINT_MODE` | `postgres` or explicit `memory` | `postgres` |
| `MEMIND_ORCHESTRATOR_DATABASE_URL` | Dedicated Orchestrator PostgreSQL URL | required |
| `MEMIND_ORCHESTRATOR_DATABASE_SCHEMA` | Checkpoint and Executor Job schema | `memind_orchestrator` |
| `MEMIND_ORCHESTRATOR_RETENTION_DAYS` | Delete terminal workflow data older than this age; `0` disables | `0` |
| `MEMIND_ORCHESTRATOR_RETENTION_SWEEP_INTERVAL_MS` | Retention sweep interval, minimum one hour | `86400000` |
| `MEMIND_ORCHESTRATOR_RETENTION_SWEEP_LIMIT` | Maximum terminal candidates per sweep | `100` |
Portal-side variables:
| Variable | Purpose | Default |
|---|---|---|
| `MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED` | Construct and schedule the Shadow observer | `false` |
| `MEMIND_ORCHESTRATOR_SHADOW_MAX_CONCURRENCY` | Maximum simultaneous observations | `2` |
| `MEMIND_ORCHESTRATOR_SHADOW_MAX_QUEUE` | Maximum waiting observations before skip | `100` |
| `MEMIND_ORCHESTRATOR_ALLOWED_ORIGINS` | Additional exact non-loopback Orchestrator origins | empty |
An Orchestrator bound beyond loopback refuses to start without the service
token. Execution refuses to start without both service and worker tokens.
Retention is disabled by default; the maintenance endpoint is dry-run unless
the authenticated caller sends `apply=true`.
## Extraction test