docs: finalize orchestrator phase 5 operations

docs/architecture/memind-orchestrator-boundary.md

@@ -54,19 +54,19 @@ Native selection.
 The initial workflow allowlist contains only `code-run-v1`. Ordinary chat and
 existing Goosed session traffic remain outside the Orchestrator path.
 
-Phase 2 implements only the `off` and `shadow` execution semantics. Although
+Phase 2 implemented only the `off` and `shadow` execution semantics. Canary and
-Canary and Active can be configured and evaluated by the control plane, the
+Active were initially planning-only so an administrative configuration mistake
-Portal does not hand execution ownership to LangGraph yet. This prevents an
+could not create two task executors.
-administrative configuration mistake from creating two task executors.
 
 Phase 3 makes that restriction explicit in the contract. Routing returns both
 `candidateEngine` and `engine`: Canary or Active may nominate LangGraph, while
 `engine` remains Native. The decision is projected as
 `workflow_execution_planned` with `dryRun=true` and `handoffAllowed=false`.
 The execution adapter normalizes authorization, idempotency, timeout,
-cancellation, and fallback controls, but its non-Native dispatch path is
+cancellation, and fallback controls. Phase 5 can transfer ownership only when
-hard-disabled in code. Changing memindadm configuration or setting an
+the memindadm switch, Portal environment gate, Orchestrator environment gate,
-environment override cannot transfer execution ownership.
+adapter enablement, worker health, authorization, and admission policy all
+agree. Any missing gate keeps Native effective.
 
 The Dry-run projection records every code-run decision while the configured
 mode is Shadow, Canary, or Active, including `canary_not_selected`. In Shadow,
@@ -96,11 +96,11 @@ authorization and side-effect policy, timeout, cancellation, fallback, and an
 idempotency key. The store interface requires atomic create-if-absent semantics
 so retries return the original job and conflicting payloads fail closed.
 
-Goosed, Aider, and OpenHands currently appear only as disabled
+Goosed, Aider, and OpenHands began as disabled `contract-only` adapters. The
-`contract-only` adapters. The reference in-memory store is non-durable and is
+reference in-memory store remains non-durable and is not a deployment backend.
-not a deployment backend. The compile-time dispatch capability remains false,
+In the current Phase 5 implementation the dispatch capability exists, but
-so creating a job records `blocked` state without importing an executor SDK,
+execution and every adapter remain disabled by default. Shadow jobs still record
-launching a process, or touching a workspace.
+terminal `blocked` state without launching a process or touching a workspace.
 
 Phase 3.3 persists that state in the Orchestrator-owned PostgreSQL database.
 LangGraph checkpoint tables and `executor_jobs` share the isolated
@@ -132,10 +132,11 @@ GET /v1/executor-jobs/:jobId
 GET /v1/executor-jobs/:jobId/events?after=<cursor>&limit=<limit>
 ```
 
-There is no HTTP submit, retry, claim, or executor cancel endpoint in this
+Phase 3.4 had no HTTP submit, retry, claim, or executor cancel endpoint.
-phase. memindadm resolves the job reference stored in LangGraph plan state and
+Phase 3.5 adds a separately authenticated worker protocol. The control-plane
-shows the job snapshot and events beside Native and graph state. The payload
+read projection still contains no task instruction, secret, absolute path, lease
-contains no task instruction, secret, absolute path, or executor SDK object.
+token, or executor SDK object; only a successful worker claim receives the
+normalized request and lease token.
 
 The implemented internal API is:
 
@@ -153,16 +154,44 @@ External state changes must use an idempotency key derived from run, graph
 version, node, and attempt. Graph state stores resource references and decisions,
 not API keys, large logs, binary artifacts, or absolute production paths.
 
-The Phase 2 graph has three deterministic nodes:
+The graph keeps three deterministic control-plane nodes:
 
 ```text
 validate_run -> build_plan -> finalize_run
 ```
 
-It requires `policy.executionMode=observe-only` and
+Observe-only runs require `sideEffectsAllowed=false`, project the Native
-`policy.sideEffectsAllowed=false`. It projects the Native executor boundary and
+boundary, and persist a blocked Executor Job. Explicit active runs require
-persists a blocked Executor Job, but cannot dispatch Goosed, Aider, OpenHands,
+`sideEffectsAllowed=true` plus every execution gate; they queue a job and enter
-Tool Gateway, or the filesystem.
+`waiting` until the worker-owned job reaches a terminal state. LangGraph never
+mounts a workspace or imports an executor SDK.
+
+## Phase 5 worker and loop boundary
+
+The task loop is split across durable owners:
+
+```text
+LangGraph run: validate -> queue -> wait -> project terminal
+Executor Job: queued -> leased -> running -> terminal/retryable
+Worker: claim -> heartbeat -> adapter -> artifact references
+```
+
+PostgreSQL row locks and lease tokens fence duplicate claims and late worker
+updates. Expired leases recover to `retryable` until `maxAttempts` is exhausted,
+then become `timed_out`. Worker heartbeats are persisted independently of job
+heartbeats so readiness and autoscaling signals also work while the queue is
+empty.
+
+Goosed is a remote HTTP/SSE adapter. Aider and OpenHands are process adapters
+intended to run in separate worker containers with workspace aliases, an
+allowlisted writable root, no shell, bounded environment/output, resource
+limits, and no Docker socket. All adapter outputs cross the boundary as bounded
+events and artifact references.
+
+Admission is defense in depth: tenant/user/workspace allowlists, global and
+per-subject concurrency, and per-minute limits are evaluated before queueing.
+The public metrics endpoint exposes queue states, claimable jobs, expired
+leases, worker health, and execution-gate state without task contents.
 
 ## Failure isolation
 

services/orchestrator/README.md

@@ -72,7 +72,7 @@ window and never changes the runtime mode. The first gate requires:
 Passing the gate means only `manual_canary_review`; it does not authorize or
 activate Canary routing.
 
-Phase 3 adds the execution handoff boundary without enabling it:
+Phase 3 introduced the execution handoff boundary in a hard-disabled state:
 
 - `workflow-execution-request-v1` normalizes authorization, timeout,
   cancellation, fallback, and idempotency controls;
@@ -82,8 +82,8 @@ Phase 3 adds the execution handoff boundary without enabling it:
   events;
 - the effective engine remains Native and non-Native dispatch throws
   `WORKFLOW_EXECUTION_HANDOFF_DISABLED`;
-- the implementation capability is hard-disabled in code, so memindadm
+- the implementation capability was hard-disabled in code while the contracts
-  configuration cannot accidentally unlock execution.
+  and observability matured.
 
 Phase 3.1 adds a Memind-owned Dry-run projection. Shadow mode continues its
 LangGraph observation while also evaluating the configured Canary allowlist and
@@ -103,12 +103,13 @@ Phase 3.2 establishes the Executor Gateway boundary:
   operations;
 - timeout, cancellation, fallback, authorization, network, and side-effect
   controls are normalized before adapter selection;
-- Goosed, Aider, and OpenHands are registered as `contract-only` adapters;
+- Goosed, Aider, and OpenHands were initially registered as `contract-only`
+  adapters;
 - the reference memory store is explicitly non-durable and used only for local
   contract tests.
 
-`EXECUTOR_DISPATCH_IMPLEMENTED=false` is a code-level gate. Phase 3.2 can record
+Phase 3.2 could record blocked, idempotent job state but could not invoke an
-blocked, idempotent job state but cannot invoke an adapter or launch a process.
+adapter or launch a process.
 
 Phase 3.3 makes that blocked state durable:
 
@@ -122,9 +123,8 @@ Phase 3.3 makes that blocked state durable:
   independently;
 - memory mode keeps both stores explicitly non-durable.
 
-The persisted job is still a preview terminal state:
+The Phase 3.3 persisted job was a preview terminal state. Existing Shadow runs
-`status=blocked`, `reason=executor_dispatch_not_implemented`, and
+remain blocked and non-executable after later phases.
-`executionEnabled=false`. Phase 3.3 does not call Goosed, Aider, or OpenHands.
 
 Phase 3.4 adds read-only Executor Job observability:
 
@@ -144,12 +144,25 @@ Phase 3.4 adds read-only Executor Job observability:
 - Canary readiness now requires both checkpoint and Executor Job storage to be
   durable.
 
-These endpoints are observational only. They do not add submit, retry, claim,
+Phase 3.5 through Phase 5 add the opt-in execution plane:
-or adapter-dispatch operations.
 
-The default memindadm mode remains `off`. Canary and Active routing are not
+- `executor-worker-v1` provides atomic claim, lease fencing, heartbeat, progress,
-wired to the LangGraph executor in Phase 3. Native Agent Run remains the sole
+  retry, timeout recovery, completion, cancellation, and drain semantics;
-executor in every mode.
+- PostgreSQL stores worker heartbeats and exposes queue/worker metrics;
+- Goosed uses the existing session HTTP/SSE transport with abort propagation;
+- Aider and OpenHands run as separate worker processes or containers with
+  workspace aliases, isolated roots, no shell, bounded environment/output, and
+  artifact references;
+- LangGraph active runs queue an Executor Job, wait durably, and converge their
+  terminal state from the job/event boundary;
+- admission policy enforces tenant/user/workspace allowlists, concurrency, and
+  per-minute limits before a job can enter the queue;
+- memindadm, Portal environment, Orchestrator environment, adapter enablement,
+  worker health, authorization, and admission are independent fail-closed gates.
+
+The default memindadm switch and both environment execution gates remain off.
+Native Agent Run therefore remains the executor until an explicitly approved
+Canary satisfies every gate.
 
 ## Admin configuration
 
@@ -159,10 +172,11 @@ Supported modes:
 
 - `off`: Native Agent Run only.
 - `shadow`: Native executes; the configured engine may observe.
-- `canary`: user allowlist and deterministic rollout percentage produce
+- `canary`: user allowlist and deterministic rollout percentage select eligible
-  Dry-run candidate decisions.
+  runs; execution still requires the explicit execution switch and environment
-- `active`: workflow-allowlisted tasks produce Dry-run primary-engine
+  gates.
-  decisions.
+- `active`: workflow-allowlisted tasks select the primary engine, subject to
+  the same health, policy, worker, and kill-switch gates.
 
 `MEMIND_ORCHESTRATOR_KILL_SWITCH=1` always forces Native selection.
 
@@ -200,6 +214,15 @@ the Orchestrator API:
 ```text
 GET /v1/executor-jobs/:jobId
 GET /v1/executor-jobs/:jobId/events?after=<cursor>&limit=<limit>
+POST /v1/executor-jobs
+POST /v1/workers/claim
+POST /v1/workers/jobs/:jobId/start
+POST /v1/workers/jobs/:jobId/heartbeat
+POST /v1/workers/jobs/:jobId/progress
+POST /v1/workers/jobs/:jobId/complete
+POST /v1/workers/jobs/:jobId/fail
+POST /v1/workers/recover-expired
+GET /v1/workers/stats
 ```
 
 ## Colima deployment
@@ -247,6 +270,11 @@ requires the repository release gates and a separately approved deployment.
 | `MEMIND_ORCHESTRATOR_HOST` | HTTP bind address | `127.0.0.1` |
 | `MEMIND_ORCHESTRATOR_PORT` | HTTP port | `8093` |
 | `MEMIND_ORCHESTRATOR_SERVICE_TOKEN` | Bearer token for `/v1/*` | empty |
+| `MEMIND_ORCHESTRATOR_WORKER_TOKEN` | Additional worker-protocol token | empty |
+| `MEMIND_ORCHESTRATOR_EXECUTION_ENABLED` | Queue admission execution gate | `false` |
+| `MEMIND_ORCHESTRATOR_ENABLED_EXECUTORS` | Enabled adapter IDs | empty |
+| `MEMIND_ORCHESTRATOR_*_ALLOWLIST` | Tenant/user/workspace admission scopes | empty / workspace alias |
+| `MEMIND_ORCHESTRATOR_MAX_*` | Global, tenant, user and rate quotas | bounded safe defaults |
 | `MEMIND_ORCHESTRATOR_CHECKPOINT_MODE` | `postgres` or explicit `memory` | `postgres` |
 | `MEMIND_ORCHESTRATOR_DATABASE_URL` | Dedicated Orchestrator PostgreSQL URL | required |
 | `MEMIND_ORCHESTRATOR_DATABASE_SCHEMA` | Checkpoint and Executor Job schema | `memind_orchestrator` |

services/orchestrator/app.test.mjs

@@ -50,6 +50,9 @@ test('orchestrator HTTP API exposes health and authenticated run endpoints', asy
   const ready = await fetch(`${server.baseUrl}/ready`);
   assert.equal(ready.status, 200);
   assert.equal((await ready.json()).ready, true);
+  const metrics = await fetch(`${server.baseUrl}/metrics`);
+  assert.equal(metrics.status, 200);
+  assert.match(await metrics.text(), /memind_orchestrator_execution_enabled 0/);
 
   const unauthorized = await fetch(`${server.baseUrl}/v1/runs`, {
     method: 'POST',

services/orchestrator/executor-job-store.test.mjs

@@ -52,6 +52,8 @@ test('PostgreSQL Executor Job Store creates an isolated schema and job table', a
   assert.match(queries[3].sql, /"memind_orchestrator_test"\."executor_job_events"/);
   assert.match(queries[3].sql, /UNIQUE \(job_id, sequence\)/);
   assert.match(queries[5].sql, /executor_job_snapshot_imported/);
+  assert.match(queries[6].sql, /executor_workers/);
+  assert.match(queries[7].sql, /executor_workers_last_seen_idx/);
 });
 
 test('PostgreSQL Executor Job Store atomically creates and reloads idempotent state', async () => {

services/orchestrator/package.json

@@ -8,6 +8,7 @@
   },
   "scripts": {
     "start": "node server.mjs",
+    "start:worker": "node worker.mjs",
     "test": "node --test *.test.mjs"
   },
   "dependencies": {