memind/services/orchestrator/README.md

# Memind Workflow Orchestrator

This directory is the extraction boundary for Memind's durable workflow runtime.
It is intentionally source-colocated with Memind while remaining framework-neutral
at its public edge.

## Dependency rule

Allowed:

```text
Memind -> orchestrator contracts / client
Orchestrator -> contracts / executor adapters
Executor adapters -> versioned internal APIs
```

Forbidden:

```text
Orchestrator -> server.mjs
Orchestrator -> user-auth.mjs
Orchestrator -> Memind business tables
Orchestrator -> production filesystem paths
Memind -> LangGraph StateGraph / Command / checkpoint types
```

LangGraph is an implementation of the generic workflow engine contract. The
Memind-facing protocol is `orchestrator-run-v1` plus `orchestrator-event-v1`.

## Current stage

Phase 1 established:

- framework-neutral run and event contracts;
- a plug-in workflow engine registry;
- a remote workflow engine client;
- versioned admin configuration;
- deterministic Off / Shadow / Canary / Active routing decisions;
- an environment-level emergency kill switch.

Phase 2 adds:

- a real LangGraph `StateGraph` service on the versioned HTTP boundary;
- a PostgreSQL `PostgresSaver` checkpoint owner;
- an explicitly non-durable MemorySaver option for tests and local debugging;
- a `code-run-v1` observe-only graph with no executor dispatch or tool access;
- a Portal fire-and-forget shadow projection;
- live health probing from memindadm;
- a separate Colima/Compose deployment artifact.

Phase 2.5 adds a Memind-owned Shadow observability projection:

- success/failure totals and rates;
- P50/P95 observation latency;
- Native terminal status beside the Shadow result;
- recent failure codes and messages;
- per-run LangGraph checkpoint and node-event inspection through the versioned
  remote API.

The projection reads `h5_agent_run_events` from the Memind control plane.
LangGraph still has no access to Memind business tables.

Phase 2.6 adds a read-only Canary readiness gate. It evaluates a fixed seven-day
window and never changes the runtime mode. The first gate requires:

- at least 20 non-smoke observations across at least five real sessions;
- at least 95% Shadow success, latency coverage, and Native terminal coverage;
- Shadow P95 latency no greater than two seconds;
- a healthy observe-only service backed by a durable checkpoint;
- a fresh, uncapped metric window while the control plane remains in Shadow.

Passing the gate means only `manual_canary_review`; it does not authorize or
activate Canary routing.

Phase 3 introduced the execution handoff boundary in a hard-disabled state:

- `workflow-execution-request-v1` normalizes authorization, timeout,
  cancellation, fallback, and idempotency controls;
- `workflow-execution-decision-v1` separates the candidate engine from the
  effective executor;
- Canary and Active selections emit auditable `workflow_execution_planned`
  events;
- the effective engine remains Native and non-Native dispatch throws
  `WORKFLOW_EXECUTION_HANDOFF_DISABLED`;
- the implementation capability was hard-disabled in code while the contracts
  and observability matured.

Phase 3.1 adds a Memind-owned Dry-run projection. Shadow mode continues its
LangGraph observation while also evaluating the configured Canary allowlist and
percentage. It records both selected and non-selected decisions so the
candidate selection rate has a complete denominator. memindadm shows candidate
reasons, task-type distribution, Native terminal coverage, recent decisions,
and an explicit alert if any stored decision ever reports
`handoffAllowed=true`.

Phase 3.2 establishes the Executor Gateway boundary:

- `executor-job-request-v1`, `executor-job-state-v1`, and
  `executor-dispatch-decision-v1` remain independent of Goosed, Aider,
  OpenHands, and LangGraph SDK types;
- jobs use workspace and artifact references rather than host filesystem paths;
- the store contract provides atomic idempotency, lookup, and state update
  operations;
- timeout, cancellation, fallback, authorization, network, and side-effect
  controls are normalized before adapter selection;
- Goosed, Aider, and OpenHands were initially registered as `contract-only`
  adapters;
- the reference memory store is explicitly non-durable and used only for local
  contract tests.

Phase 3.2 could record blocked, idempotent job state but could not invoke an
adapter or launch a process.

Phase 3.3 makes that blocked state durable:

- the Orchestrator owns a PostgreSQL `executor_jobs` table beside its LangGraph
  checkpoint tables, under the same isolated schema and database credentials;
- the `build_plan` node atomically creates one deterministic Executor Job per
  run and graph-node version;
- retries recover the same job through its idempotency key, while a changed
  semantic request fails closed;
- `/health` and `/ready` report and probe checkpoint and Executor Job storage
  independently;
- memory mode keeps both stores explicitly non-durable.

The Phase 3.3 persisted job was a preview terminal state. Existing Shadow runs
remain blocked and non-executable after later phases.

Phase 3.4 adds read-only Executor Job observability:

- `executor-job-event-v1` is stored in the separate PostgreSQL
  `executor_job_events` table with a per-job sequence cursor;
- initial state and its first event are committed atomically, as are later
  state transitions and their events;
- existing Phase 3.3 snapshots receive a deterministic
  `executor_job_snapshot_imported` migration event;
- the versioned API exposes job state and cursor-paged events without exposing
  task instructions, credentials, workspace paths, or executor SDK types;
- unsafe or oversized run identifiers are represented by a deterministic
  SHA-256-derived opaque Executor Job id so PostgreSQL and URL identifiers stay
  bounded;
- Shadow Run detail in memindadm joins Native state, LangGraph checkpoints,
  Executor Job state, and Executor Job events;
- Canary readiness now requires both checkpoint and Executor Job storage to be
  durable.

Phase 3.5 through Phase 5 add the opt-in execution plane:

- `executor-worker-v1` provides atomic claim, lease fencing, heartbeat, progress,
  retry, timeout recovery, completion, cancellation, and drain semantics;
- PostgreSQL stores worker heartbeats and exposes queue/worker metrics;
- Goosed uses the existing session HTTP/SSE transport with abort propagation;
- Aider and OpenHands run as separate worker processes or containers with
  workspace aliases, isolated roots, no shell, bounded environment/output, and
  artifact references;
- LangGraph active runs queue an Executor Job, wait durably, and converge their
  terminal state from the job/event boundary;
- admission policy enforces tenant/user/workspace allowlists, concurrency, and
  per-minute limits before a job can enter the queue;
- memindadm, Portal environment, Orchestrator environment, adapter enablement,
  worker health, authorization, and admission are independent fail-closed gates.

The default memindadm switch and both environment execution gates remain off.
Native Agent Run therefore remains the executor until an explicitly approved
Canary satisfies every gate.

## Admin configuration

The memindadm page is `/ops/admin/orchestrator`.

Supported modes:

- `off`: Native Agent Run only.
- `shadow`: Native executes; the configured engine may observe.
- `canary`: user allowlist and deterministic rollout percentage select eligible
  runs; execution still requires the explicit execution switch and environment
  gates.
- `active`: workflow-allowlisted tasks select the primary engine, subject to
  the same health, policy, worker, and kill-switch gates.

`MEMIND_ORCHESTRATOR_KILL_SWITCH=1` always forces Native selection.

## Local process

PostgreSQL is the default and required checkpoint and Executor Job backend:

```bash
MEMIND_ORCHESTRATOR_DATABASE_URL='postgresql://...' \
MEMIND_ORCHESTRATOR_SERVICE_TOKEN='...' \
pnpm dev:orchestrator
```

Memory mode is intentionally opt-in and non-durable:

```bash
MEMIND_ORCHESTRATOR_CHECKPOINT_MODE=memory pnpm dev:orchestrator
```

The service binds to `127.0.0.1:8093` by default. Configure the same URL and
service token in Portal, set
`MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=1` on that Portal instance,
then enable `shadow` in `/ops/admin/orchestrator`. Without the Portal gate,
Native code runs do not construct or call the observer.

The admin runtime projection distinguishes requested Shadow mode from effective
Portal wiring. Canary readiness requires `shadow_wiring_enabled=true`; a closed
environment gate therefore blocks promotion even when older successful samples
are still present.

memindadm exposes the projection through:

```text
GET /admin-api/orchestrator/shadow-runs
GET /admin-api/orchestrator/shadow-runs/:runId
GET /admin-api/orchestrator/canary-readiness
GET /admin-api/orchestrator/execution-plans
```

Selecting a Shadow Run in memindadm also reads the linked Executor Job through
the Orchestrator API:

```text
GET /v1/executor-jobs/:jobId
GET /v1/executor-jobs/:jobId/events?after=<cursor>&limit=<limit>
POST /v1/executor-jobs
POST /v1/workers/claim
POST /v1/workers/jobs/:jobId/start
POST /v1/workers/jobs/:jobId/heartbeat
POST /v1/workers/jobs/:jobId/progress
POST /v1/workers/jobs/:jobId/complete
POST /v1/workers/jobs/:jobId/fail
POST /v1/workers/recover-expired
GET /v1/workers/stats
DELETE /v1/runs/:runId
POST /v1/maintenance/purge-terminal-runs
```

Shadow RunSpecs use the `control-plane-only-v1` policy and do not copy user
messages, user IDs, or session IDs into Orchestrator storage. Deleting a
terminal run removes its checkpoint thread and linked terminal Executor Job.

## Colima deployment

Colima is the recommended first container host on macOS because this service and
its PostgreSQL checkpoint database have an isolated Compose lifecycle:

```bash
cd deploy/orchestrator
cp .env.example .env
# Replace both example secrets before starting.
docker compose up -d --build
docker compose ps
curl http://127.0.0.1:8093/ready
```

When Portal or the Shadow smoke process runs outside Compose, export the same
service token into the child process environment. Plain `source` does not export
new shell variables by itself:

```bash
set -a
source deploy/orchestrator/.env
set +a
export MEMIND_ORCHESTRATOR_URL=http://127.0.0.1:8093
pnpm dev
```

The Compose project:

- publishes only the Orchestrator port on loopback;
- does not publish PostgreSQL;
- does not mount the Docker socket or a Memind workspace;
- keeps PostgreSQL on a dedicated internal network while the service also joins
  a separate edge network for its loopback-published port;
- is not part of the Goosed Compose lifecycle.

This local Compose artifact is not a production release path. Production still
requires the repository release gates and a separately approved deployment.

## Runtime environment

| Variable | Purpose | Default |
|---|---|---|
| `MEMIND_ORCHESTRATOR_HOST` | HTTP bind address | `127.0.0.1` |
| `MEMIND_ORCHESTRATOR_PORT` | HTTP port | `8093` |
| `MEMIND_ORCHESTRATOR_SERVICE_TOKEN` | Bearer token for `/v1/*` | empty |
| `MEMIND_ORCHESTRATOR_WORKER_TOKEN` | Additional worker-protocol token | empty |
| `MEMIND_ORCHESTRATOR_EXECUTION_ENABLED` | Queue admission execution gate | `false` |
| `MEMIND_ORCHESTRATOR_ENABLED_EXECUTORS` | Enabled adapter IDs | empty |
| `MEMIND_ORCHESTRATOR_*_ALLOWLIST` | Tenant/user/workspace admission scopes | empty / workspace alias |
| `MEMIND_ORCHESTRATOR_MAX_*` | Global, tenant, user and rate quotas | bounded safe defaults |
| `MEMIND_ORCHESTRATOR_CHECKPOINT_MODE` | `postgres` or explicit `memory` | `postgres` |
| `MEMIND_ORCHESTRATOR_DATABASE_URL` | Dedicated Orchestrator PostgreSQL URL | required |
| `MEMIND_ORCHESTRATOR_DATABASE_SCHEMA` | Checkpoint and Executor Job schema | `memind_orchestrator` |
| `MEMIND_ORCHESTRATOR_RETENTION_DAYS` | Delete terminal workflow data older than this age; `0` disables | `0` |
| `MEMIND_ORCHESTRATOR_RETENTION_SWEEP_INTERVAL_MS` | Retention sweep interval, minimum one hour | `86400000` |
| `MEMIND_ORCHESTRATOR_RETENTION_SWEEP_LIMIT` | Maximum terminal candidates per sweep | `100` |

Portal-side variables:

| Variable | Purpose | Default |
|---|---|---|
| `MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED` | Construct and schedule the Shadow observer | `false` |
| `MEMIND_ORCHESTRATOR_SHADOW_MAX_CONCURRENCY` | Maximum simultaneous observations | `2` |
| `MEMIND_ORCHESTRATOR_SHADOW_MAX_QUEUE` | Maximum waiting observations before skip | `100` |
| `MEMIND_ORCHESTRATOR_ALLOWED_ORIGINS` | Additional exact non-loopback Orchestrator origins | empty |

An Orchestrator bound beyond loopback refuses to start without the service
token. Execution refuses to start without both service and worker tokens.
Retention is disabled by default; the maintenance endpoint is dry-run unless
the authenticated caller sends `apply=true`.

## Extraction test

The service is ready to move into a separate repository only when:

1. it can run against a mock Memind control plane;
2. it owns its checkpoint database and credentials;
3. task inputs use resource references rather than host paths;
4. executor calls use versioned adapters;
5. moving the service requires only changing `ORCHESTRATOR_URL`.