diff --git a/docs/architecture/memind-orchestrator-boundary.md b/docs/architecture/memind-orchestrator-boundary.md index 7923c43..d9c6960 100644 --- a/docs/architecture/memind-orchestrator-boundary.md +++ b/docs/architecture/memind-orchestrator-boundary.md @@ -54,19 +54,19 @@ Native selection. The initial workflow allowlist contains only `code-run-v1`. Ordinary chat and existing Goosed session traffic remain outside the Orchestrator path. -Phase 2 implements only the `off` and `shadow` execution semantics. Although -Canary and Active can be configured and evaluated by the control plane, the -Portal does not hand execution ownership to LangGraph yet. This prevents an -administrative configuration mistake from creating two task executors. +Phase 2 implemented only the `off` and `shadow` execution semantics. Canary and +Active were initially planning-only so an administrative configuration mistake +could not create two task executors. Phase 3 makes that restriction explicit in the contract. Routing returns both `candidateEngine` and `engine`: Canary or Active may nominate LangGraph, while `engine` remains Native. The decision is projected as `workflow_execution_planned` with `dryRun=true` and `handoffAllowed=false`. The execution adapter normalizes authorization, idempotency, timeout, -cancellation, and fallback controls, but its non-Native dispatch path is -hard-disabled in code. Changing memindadm configuration or setting an -environment override cannot transfer execution ownership. +cancellation, and fallback controls. Phase 5 can transfer ownership only when +the memindadm switch, Portal environment gate, Orchestrator environment gate, +adapter enablement, worker health, authorization, and admission policy all +agree. Any missing gate keeps Native effective. The Dry-run projection records every code-run decision while the configured mode is Shadow, Canary, or Active, including `canary_not_selected`. In Shadow, @@ -96,11 +96,11 @@ authorization and side-effect policy, timeout, cancellation, fallback, and an idempotency key. The store interface requires atomic create-if-absent semantics so retries return the original job and conflicting payloads fail closed. -Goosed, Aider, and OpenHands currently appear only as disabled -`contract-only` adapters. The reference in-memory store is non-durable and is -not a deployment backend. The compile-time dispatch capability remains false, -so creating a job records `blocked` state without importing an executor SDK, -launching a process, or touching a workspace. +Goosed, Aider, and OpenHands began as disabled `contract-only` adapters. The +reference in-memory store remains non-durable and is not a deployment backend. +In the current Phase 5 implementation the dispatch capability exists, but +execution and every adapter remain disabled by default. Shadow jobs still record +terminal `blocked` state without launching a process or touching a workspace. Phase 3.3 persists that state in the Orchestrator-owned PostgreSQL database. LangGraph checkpoint tables and `executor_jobs` share the isolated @@ -132,10 +132,11 @@ GET /v1/executor-jobs/:jobId GET /v1/executor-jobs/:jobId/events?after=&limit= ``` -There is no HTTP submit, retry, claim, or executor cancel endpoint in this -phase. memindadm resolves the job reference stored in LangGraph plan state and -shows the job snapshot and events beside Native and graph state. The payload -contains no task instruction, secret, absolute path, or executor SDK object. +Phase 3.4 had no HTTP submit, retry, claim, or executor cancel endpoint. +Phase 3.5 adds a separately authenticated worker protocol. The control-plane +read projection still contains no task instruction, secret, absolute path, lease +token, or executor SDK object; only a successful worker claim receives the +normalized request and lease token. The implemented internal API is: @@ -153,16 +154,44 @@ External state changes must use an idempotency key derived from run, graph version, node, and attempt. Graph state stores resource references and decisions, not API keys, large logs, binary artifacts, or absolute production paths. -The Phase 2 graph has three deterministic nodes: +The graph keeps three deterministic control-plane nodes: ```text validate_run -> build_plan -> finalize_run ``` -It requires `policy.executionMode=observe-only` and -`policy.sideEffectsAllowed=false`. It projects the Native executor boundary and -persists a blocked Executor Job, but cannot dispatch Goosed, Aider, OpenHands, -Tool Gateway, or the filesystem. +Observe-only runs require `sideEffectsAllowed=false`, project the Native +boundary, and persist a blocked Executor Job. Explicit active runs require +`sideEffectsAllowed=true` plus every execution gate; they queue a job and enter +`waiting` until the worker-owned job reaches a terminal state. LangGraph never +mounts a workspace or imports an executor SDK. + +## Phase 5 worker and loop boundary + +The task loop is split across durable owners: + +```text +LangGraph run: validate -> queue -> wait -> project terminal +Executor Job: queued -> leased -> running -> terminal/retryable +Worker: claim -> heartbeat -> adapter -> artifact references +``` + +PostgreSQL row locks and lease tokens fence duplicate claims and late worker +updates. Expired leases recover to `retryable` until `maxAttempts` is exhausted, +then become `timed_out`. Worker heartbeats are persisted independently of job +heartbeats so readiness and autoscaling signals also work while the queue is +empty. + +Goosed is a remote HTTP/SSE adapter. Aider and OpenHands are process adapters +intended to run in separate worker containers with workspace aliases, an +allowlisted writable root, no shell, bounded environment/output, resource +limits, and no Docker socket. All adapter outputs cross the boundary as bounded +events and artifact references. + +Admission is defense in depth: tenant/user/workspace allowlists, global and +per-subject concurrency, and per-minute limits are evaluated before queueing. +The public metrics endpoint exposes queue states, claimable jobs, expired +leases, worker health, and execution-gate state without task contents. ## Failure isolation diff --git a/services/orchestrator/README.md b/services/orchestrator/README.md index baf60ff..861a637 100644 --- a/services/orchestrator/README.md +++ b/services/orchestrator/README.md @@ -72,7 +72,7 @@ window and never changes the runtime mode. The first gate requires: Passing the gate means only `manual_canary_review`; it does not authorize or activate Canary routing. -Phase 3 adds the execution handoff boundary without enabling it: +Phase 3 introduced the execution handoff boundary in a hard-disabled state: - `workflow-execution-request-v1` normalizes authorization, timeout, cancellation, fallback, and idempotency controls; @@ -82,8 +82,8 @@ Phase 3 adds the execution handoff boundary without enabling it: events; - the effective engine remains Native and non-Native dispatch throws `WORKFLOW_EXECUTION_HANDOFF_DISABLED`; -- the implementation capability is hard-disabled in code, so memindadm - configuration cannot accidentally unlock execution. +- the implementation capability was hard-disabled in code while the contracts + and observability matured. Phase 3.1 adds a Memind-owned Dry-run projection. Shadow mode continues its LangGraph observation while also evaluating the configured Canary allowlist and @@ -103,12 +103,13 @@ Phase 3.2 establishes the Executor Gateway boundary: operations; - timeout, cancellation, fallback, authorization, network, and side-effect controls are normalized before adapter selection; -- Goosed, Aider, and OpenHands are registered as `contract-only` adapters; +- Goosed, Aider, and OpenHands were initially registered as `contract-only` + adapters; - the reference memory store is explicitly non-durable and used only for local contract tests. -`EXECUTOR_DISPATCH_IMPLEMENTED=false` is a code-level gate. Phase 3.2 can record -blocked, idempotent job state but cannot invoke an adapter or launch a process. +Phase 3.2 could record blocked, idempotent job state but could not invoke an +adapter or launch a process. Phase 3.3 makes that blocked state durable: @@ -122,9 +123,8 @@ Phase 3.3 makes that blocked state durable: independently; - memory mode keeps both stores explicitly non-durable. -The persisted job is still a preview terminal state: -`status=blocked`, `reason=executor_dispatch_not_implemented`, and -`executionEnabled=false`. Phase 3.3 does not call Goosed, Aider, or OpenHands. +The Phase 3.3 persisted job was a preview terminal state. Existing Shadow runs +remain blocked and non-executable after later phases. Phase 3.4 adds read-only Executor Job observability: @@ -144,12 +144,25 @@ Phase 3.4 adds read-only Executor Job observability: - Canary readiness now requires both checkpoint and Executor Job storage to be durable. -These endpoints are observational only. They do not add submit, retry, claim, -or adapter-dispatch operations. +Phase 3.5 through Phase 5 add the opt-in execution plane: -The default memindadm mode remains `off`. Canary and Active routing are not -wired to the LangGraph executor in Phase 3. Native Agent Run remains the sole -executor in every mode. +- `executor-worker-v1` provides atomic claim, lease fencing, heartbeat, progress, + retry, timeout recovery, completion, cancellation, and drain semantics; +- PostgreSQL stores worker heartbeats and exposes queue/worker metrics; +- Goosed uses the existing session HTTP/SSE transport with abort propagation; +- Aider and OpenHands run as separate worker processes or containers with + workspace aliases, isolated roots, no shell, bounded environment/output, and + artifact references; +- LangGraph active runs queue an Executor Job, wait durably, and converge their + terminal state from the job/event boundary; +- admission policy enforces tenant/user/workspace allowlists, concurrency, and + per-minute limits before a job can enter the queue; +- memindadm, Portal environment, Orchestrator environment, adapter enablement, + worker health, authorization, and admission are independent fail-closed gates. + +The default memindadm switch and both environment execution gates remain off. +Native Agent Run therefore remains the executor until an explicitly approved +Canary satisfies every gate. ## Admin configuration @@ -159,10 +172,11 @@ Supported modes: - `off`: Native Agent Run only. - `shadow`: Native executes; the configured engine may observe. -- `canary`: user allowlist and deterministic rollout percentage produce - Dry-run candidate decisions. -- `active`: workflow-allowlisted tasks produce Dry-run primary-engine - decisions. +- `canary`: user allowlist and deterministic rollout percentage select eligible + runs; execution still requires the explicit execution switch and environment + gates. +- `active`: workflow-allowlisted tasks select the primary engine, subject to + the same health, policy, worker, and kill-switch gates. `MEMIND_ORCHESTRATOR_KILL_SWITCH=1` always forces Native selection. @@ -200,6 +214,15 @@ the Orchestrator API: ```text GET /v1/executor-jobs/:jobId GET /v1/executor-jobs/:jobId/events?after=&limit= +POST /v1/executor-jobs +POST /v1/workers/claim +POST /v1/workers/jobs/:jobId/start +POST /v1/workers/jobs/:jobId/heartbeat +POST /v1/workers/jobs/:jobId/progress +POST /v1/workers/jobs/:jobId/complete +POST /v1/workers/jobs/:jobId/fail +POST /v1/workers/recover-expired +GET /v1/workers/stats ``` ## Colima deployment @@ -247,6 +270,11 @@ requires the repository release gates and a separately approved deployment. | `MEMIND_ORCHESTRATOR_HOST` | HTTP bind address | `127.0.0.1` | | `MEMIND_ORCHESTRATOR_PORT` | HTTP port | `8093` | | `MEMIND_ORCHESTRATOR_SERVICE_TOKEN` | Bearer token for `/v1/*` | empty | +| `MEMIND_ORCHESTRATOR_WORKER_TOKEN` | Additional worker-protocol token | empty | +| `MEMIND_ORCHESTRATOR_EXECUTION_ENABLED` | Queue admission execution gate | `false` | +| `MEMIND_ORCHESTRATOR_ENABLED_EXECUTORS` | Enabled adapter IDs | empty | +| `MEMIND_ORCHESTRATOR_*_ALLOWLIST` | Tenant/user/workspace admission scopes | empty / workspace alias | +| `MEMIND_ORCHESTRATOR_MAX_*` | Global, tenant, user and rate quotas | bounded safe defaults | | `MEMIND_ORCHESTRATOR_CHECKPOINT_MODE` | `postgres` or explicit `memory` | `postgres` | | `MEMIND_ORCHESTRATOR_DATABASE_URL` | Dedicated Orchestrator PostgreSQL URL | required | | `MEMIND_ORCHESTRATOR_DATABASE_SCHEMA` | Checkpoint and Executor Job schema | `memind_orchestrator` | diff --git a/services/orchestrator/app.test.mjs b/services/orchestrator/app.test.mjs index 67f6c92..d8f422f 100644 --- a/services/orchestrator/app.test.mjs +++ b/services/orchestrator/app.test.mjs @@ -50,6 +50,9 @@ test('orchestrator HTTP API exposes health and authenticated run endpoints', asy const ready = await fetch(`${server.baseUrl}/ready`); assert.equal(ready.status, 200); assert.equal((await ready.json()).ready, true); + const metrics = await fetch(`${server.baseUrl}/metrics`); + assert.equal(metrics.status, 200); + assert.match(await metrics.text(), /memind_orchestrator_execution_enabled 0/); const unauthorized = await fetch(`${server.baseUrl}/v1/runs`, { method: 'POST', diff --git a/services/orchestrator/executor-job-store.test.mjs b/services/orchestrator/executor-job-store.test.mjs index 0a99bf7..ae9c890 100644 --- a/services/orchestrator/executor-job-store.test.mjs +++ b/services/orchestrator/executor-job-store.test.mjs @@ -52,6 +52,8 @@ test('PostgreSQL Executor Job Store creates an isolated schema and job table', a assert.match(queries[3].sql, /"memind_orchestrator_test"\."executor_job_events"/); assert.match(queries[3].sql, /UNIQUE \(job_id, sequence\)/); assert.match(queries[5].sql, /executor_job_snapshot_imported/); + assert.match(queries[6].sql, /executor_workers/); + assert.match(queries[7].sql, /executor_workers_last_seen_idx/); }); test('PostgreSQL Executor Job Store atomically creates and reloads idempotent state', async () => { diff --git a/services/orchestrator/package.json b/services/orchestrator/package.json index 69a4bfe..20ac427 100644 --- a/services/orchestrator/package.json +++ b/services/orchestrator/package.json @@ -8,6 +8,7 @@ }, "scripts": { "start": "node server.mjs", + "start:worker": "node worker.mjs", "test": "node --test *.test.mjs" }, "dependencies": {