tkmind/memind
1
0
Fork 0
Code Issues Pull Requests 1 Actions Packages Projects Releases Wiki Activity

docs: finalize orchestrator phase 5 operations

Browse Source
This commit is contained in:
john
2026-07-24 23:55:40 +08:00
parent f6f2cd0933
commit e3d7b9ee76
5 changed files with 102 additions and 39 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/architecture/memind-orchestrator-boundary.md
+50 -21
View File
@@ -54,19 +54,19 @@ Native selection.
The initial workflow allowlist contains only `code-run-v1`. Ordinary chat and
existing Goosed session traffic remain outside the Orchestrator path.
Phase 2 implements only the `off` and `shadow` execution semantics. Although
Canary and Active can be configured and evaluated by the control plane, the
Portal does not hand execution ownership to LangGraph yet. This prevents an
administrative configuration mistake from creating two task executors.
Phase 2 implemented only the `off` and `shadow` execution semantics. Canary and
Active were initially planning-only so an administrative configuration mistake
could not create two task executors.
Phase 3 makes that restriction explicit in the contract. Routing returns both
`candidateEngine` and `engine`: Canary or Active may nominate LangGraph, while
`engine` remains Native. The decision is projected as
`workflow_execution_planned` with `dryRun=true` and `handoffAllowed=false`.
The execution adapter normalizes authorization, idempotency, timeout,
cancellation, and fallback controls, but its non-Native dispatch path is
hard-disabled in code. Changing memindadm configuration or setting an
environment override cannot transfer execution ownership.
cancellation, and fallback controls. Phase 5 can transfer ownership only when
the memindadm switch, Portal environment gate, Orchestrator environment gate,
adapter enablement, worker health, authorization, and admission policy all
agree. Any missing gate keeps Native effective.
The Dry-run projection records every code-run decision while the configured
mode is Shadow, Canary, or Active, including `canary_not_selected`. In Shadow,
@@ -96,11 +96,11 @@ authorization and side-effect policy, timeout, cancellation, fallback, and an
idempotency key. The store interface requires atomic create-if-absent semantics
so retries return the original job and conflicting payloads fail closed.
Goosed, Aider, and OpenHands currently appear only as disabled
`contract-only` adapters. The reference in-memory store is non-durable and is
not a deployment backend. The compile-time dispatch capability remains false,
so creating a job records `blocked` state without importing an executor SDK,
launching a process, or touching a workspace.
Goosed, Aider, and OpenHands began as disabled `contract-only` adapters. The
reference in-memory store remains non-durable and is not a deployment backend.
In the current Phase 5 implementation the dispatch capability exists, but
execution and every adapter remain disabled by default. Shadow jobs still record
terminal `blocked` state without launching a process or touching a workspace.
Phase 3.3 persists that state in the Orchestrator-owned PostgreSQL database.
LangGraph checkpoint tables and `executor_jobs` share the isolated
@@ -132,10 +132,11 @@ GET /v1/executor-jobs/:jobId
GET /v1/executor-jobs/:jobId/events?after=<cursor>&limit=<limit>
```
There is no HTTP submit, retry, claim, or executor cancel endpoint in this
phase. memindadm resolves the job reference stored in LangGraph plan state and
shows the job snapshot and events beside Native and graph state. The payload
contains no task instruction, secret, absolute path, or executor SDK object.
Phase 3.4 had no HTTP submit, retry, claim, or executor cancel endpoint.
Phase 3.5 adds a separately authenticated worker protocol. The control-plane
read projection still contains no task instruction, secret, absolute path, lease
token, or executor SDK object; only a successful worker claim receives the
normalized request and lease token.
The implemented internal API is:
@@ -153,16 +154,44 @@ External state changes must use an idempotency key derived from run, graph
version, node, and attempt. Graph state stores resource references and decisions,
not API keys, large logs, binary artifacts, or absolute production paths.
The Phase 2 graph has three deterministic nodes:
The graph keeps three deterministic control-plane nodes:
```text
validate_run -> build_plan -> finalize_run
```
It requires `policy.executionMode=observe-only` and
`policy.sideEffectsAllowed=false`. It projects the Native executor boundary and
persists a blocked Executor Job, but cannot dispatch Goosed, Aider, OpenHands,
Tool Gateway, or the filesystem.
Observe-only runs require `sideEffectsAllowed=false`, project the Native
boundary, and persist a blocked Executor Job. Explicit active runs require
`sideEffectsAllowed=true` plus every execution gate; they queue a job and enter
`waiting` until the worker-owned job reaches a terminal state. LangGraph never
mounts a workspace or imports an executor SDK.
## Phase 5 worker and loop boundary
The task loop is split across durable owners:
```text
LangGraph run: validate -> queue -> wait -> project terminal
Executor Job: queued -> leased -> running -> terminal/retryable
Worker: claim -> heartbeat -> adapter -> artifact references
```
PostgreSQL row locks and lease tokens fence duplicate claims and late worker
updates. Expired leases recover to `retryable` until `maxAttempts` is exhausted,
then become `timed_out`. Worker heartbeats are persisted independently of job
heartbeats so readiness and autoscaling signals also work while the queue is
empty.
Goosed is a remote HTTP/SSE adapter. Aider and OpenHands are process adapters
intended to run in separate worker containers with workspace aliases, an
allowlisted writable root, no shell, bounded environment/output, resource
limits, and no Docker socket. All adapter outputs cross the boundary as bounded
events and artifact references.
Admission is defense in depth: tenant/user/workspace allowlists, global and
per-subject concurrency, and per-minute limits are evaluated before queueing.
The public metrics endpoint exposes queue states, claimable jobs, expired
leases, worker health, and execution-gate state without task contents.
## Failure isolation
services/orchestrator/README.md
+46 -18
View File
@@ -72,7 +72,7 @@ window and never changes the runtime mode. The first gate requires:
Passing the gate means only `manual_canary_review`; it does not authorize or
activate Canary routing.
Phase 3 adds the execution handoff boundary without enabling it:
Phase 3 introduced the execution handoff boundary in a hard-disabled state:
- `workflow-execution-request-v1` normalizes authorization, timeout,
cancellation, fallback, and idempotency controls;
@@ -82,8 +82,8 @@ Phase 3 adds the execution handoff boundary without enabling it:
events;
- the effective engine remains Native and non-Native dispatch throws
`WORKFLOW_EXECUTION_HANDOFF_DISABLED`;
- the implementation capability is hard-disabled in code, so memindadm
configuration cannot accidentally unlock execution.
- the implementation capability was hard-disabled in code while the contracts
and observability matured.
Phase 3.1 adds a Memind-owned Dry-run projection. Shadow mode continues its
LangGraph observation while also evaluating the configured Canary allowlist and
@@ -103,12 +103,13 @@ Phase 3.2 establishes the Executor Gateway boundary:
operations;
- timeout, cancellation, fallback, authorization, network, and side-effect
controls are normalized before adapter selection;
- Goosed, Aider, and OpenHands are registered as `contract-only` adapters;
- Goosed, Aider, and OpenHands were initially registered as `contract-only`
adapters;
- the reference memory store is explicitly non-durable and used only for local
contract tests.
`EXECUTOR_DISPATCH_IMPLEMENTED=false` is a code-level gate. Phase 3.2 can record
blocked, idempotent job state but cannot invoke an adapter or launch a process.
Phase 3.2 could record blocked, idempotent job state but could not invoke an
adapter or launch a process.
Phase 3.3 makes that blocked state durable:
@@ -122,9 +123,8 @@ Phase 3.3 makes that blocked state durable:
independently;
- memory mode keeps both stores explicitly non-durable.
The persisted job is still a preview terminal state:
`status=blocked`, `reason=executor_dispatch_not_implemented`, and
`executionEnabled=false`. Phase 3.3 does not call Goosed, Aider, or OpenHands.
The Phase 3.3 persisted job was a preview terminal state. Existing Shadow runs
remain blocked and non-executable after later phases.
Phase 3.4 adds read-only Executor Job observability:
@@ -144,12 +144,25 @@ Phase 3.4 adds read-only Executor Job observability:
- Canary readiness now requires both checkpoint and Executor Job storage to be
durable.
These endpoints are observational only. They do not add submit, retry, claim,
or adapter-dispatch operations.
Phase 3.5 through Phase 5 add the opt-in execution plane:
The default memindadm mode remains `off`. Canary and Active routing are not
wired to the LangGraph executor in Phase 3. Native Agent Run remains the sole
executor in every mode.
- `executor-worker-v1` provides atomic claim, lease fencing, heartbeat, progress,
retry, timeout recovery, completion, cancellation, and drain semantics;
- PostgreSQL stores worker heartbeats and exposes queue/worker metrics;
- Goosed uses the existing session HTTP/SSE transport with abort propagation;
- Aider and OpenHands run as separate worker processes or containers with
workspace aliases, isolated roots, no shell, bounded environment/output, and
artifact references;
- LangGraph active runs queue an Executor Job, wait durably, and converge their
terminal state from the job/event boundary;
- admission policy enforces tenant/user/workspace allowlists, concurrency, and
per-minute limits before a job can enter the queue;
- memindadm, Portal environment, Orchestrator environment, adapter enablement,
worker health, authorization, and admission are independent fail-closed gates.
The default memindadm switch and both environment execution gates remain off.
Native Agent Run therefore remains the executor until an explicitly approved
Canary satisfies every gate.
## Admin configuration
@@ -159,10 +172,11 @@ Supported modes:
- `off`: Native Agent Run only.
- `shadow`: Native executes; the configured engine may observe.
- `canary`: user allowlist and deterministic rollout percentage produce
Dry-run candidate decisions.
- `active`: workflow-allowlisted tasks produce Dry-run primary-engine
decisions.
- `canary`: user allowlist and deterministic rollout percentage select eligible
runs; execution still requires the explicit execution switch and environment
gates.
- `active`: workflow-allowlisted tasks select the primary engine, subject to
the same health, policy, worker, and kill-switch gates.
`MEMIND_ORCHESTRATOR_KILL_SWITCH=1` always forces Native selection.
@@ -200,6 +214,15 @@ the Orchestrator API:
```text
GET /v1/executor-jobs/:jobId
GET /v1/executor-jobs/:jobId/events?after=<cursor>&limit=<limit>
POST /v1/executor-jobs
POST /v1/workers/claim
POST /v1/workers/jobs/:jobId/start
POST /v1/workers/jobs/:jobId/heartbeat
POST /v1/workers/jobs/:jobId/progress
POST /v1/workers/jobs/:jobId/complete
POST /v1/workers/jobs/:jobId/fail
POST /v1/workers/recover-expired
GET /v1/workers/stats
```
## Colima deployment
@@ -247,6 +270,11 @@ requires the repository release gates and a separately approved deployment.
| `MEMIND_ORCHESTRATOR_HOST` | HTTP bind address | `127.0.0.1` |
| `MEMIND_ORCHESTRATOR_PORT` | HTTP port | `8093` |
| `MEMIND_ORCHESTRATOR_SERVICE_TOKEN` | Bearer token for `/v1/*` | empty |
| `MEMIND_ORCHESTRATOR_WORKER_TOKEN` | Additional worker-protocol token | empty |
| `MEMIND_ORCHESTRATOR_EXECUTION_ENABLED` | Queue admission execution gate | `false` |
| `MEMIND_ORCHESTRATOR_ENABLED_EXECUTORS` | Enabled adapter IDs | empty |
| `MEMIND_ORCHESTRATOR_*_ALLOWLIST` | Tenant/user/workspace admission scopes | empty / workspace alias |
| `MEMIND_ORCHESTRATOR_MAX_*` | Global, tenant, user and rate quotas | bounded safe defaults |
| `MEMIND_ORCHESTRATOR_CHECKPOINT_MODE` | `postgres` or explicit `memory` | `postgres` |
| `MEMIND_ORCHESTRATOR_DATABASE_URL` | Dedicated Orchestrator PostgreSQL URL | required |
| `MEMIND_ORCHESTRATOR_DATABASE_SCHEMA` | Checkpoint and Executor Job schema | `memind_orchestrator` |
services/orchestrator/app.test.mjs
+3
View File
@@ -50,6 +50,9 @@ test('orchestrator HTTP API exposes health and authenticated run endpoints', asy
const ready = await fetch(`${server.baseUrl}/ready`);
assert.equal(ready.status, 200);
assert.equal((await ready.json()).ready, true);
const metrics = await fetch(`${server.baseUrl}/metrics`);
assert.equal(metrics.status, 200);
assert.match(await metrics.text(), /memind_orchestrator_execution_enabled 0/);
const unauthorized = await fetch(`${server.baseUrl}/v1/runs`, {
method: 'POST',
services/orchestrator/executor-job-store.test.mjs
+2
View File
@@ -52,6 +52,8 @@ test('PostgreSQL Executor Job Store creates an isolated schema and job table', a
assert.match(queries[3].sql, /"memind_orchestrator_test"\."executor_job_events"/);
assert.match(queries[3].sql, /UNIQUE \(job_id, sequence\)/);
assert.match(queries[5].sql, /executor_job_snapshot_imported/);
assert.match(queries[6].sql, /executor_workers/);
assert.match(queries[7].sql, /executor_workers_last_seen_idx/);
});
test('PostgreSQL Executor Job Store atomically creates and reloads idempotent state', async () => {
services/orchestrator/package.json
+1
View File
@@ -8,6 +8,7 @@
},
"scripts": {
"start": "node server.mjs",
"start:worker": "node worker.mjs",
"test": "node --test *.test.mjs"
},
"dependencies": {