memind/docs/architecture/memind-orchestrator-boundary.md

# Memind Workflow Orchestrator Boundary

## Decision

The Orchestrator starts in the Memind repository but is designed as an
independently deployable bounded context. Source colocation does not authorize
in-process execution inside Portal.

The stable boundary is:

```text
Memind Control Plane
  -> versioned Orchestrator API / events
Workflow Orchestrator
  -> versioned Executor Gateway
Executor adapters
  -> Goosed / Aider / OpenHands
```

LangGraph remains an internal workflow engine implementation. Memind must not
depend on LangGraph graph, checkpoint, command, or interrupt types.

## Ownership

| Data | Owner |
|---|---|
| User, authorization, billing, product run | Memind |
| Workflow node state, checkpoint, interrupt | Orchestrator |
| Goosed session | Goosed plus Session Broker mapping |
| Executor job | Executor Gateway / executor adapter |
| Workspace and deliverable | Workspace / Artifact / MindSpace services |
| User-visible progress and audit | Memind run-event projection |

The Orchestrator must not read or write Memind user, billing, capability, or
provider-key tables. A shared PostgreSQL instance is acceptable during the
single-host phase, but the Orchestrator must use its own database and database
credentials.

## Runtime modes

memindadm owns the versioned routing configuration:

| Mode | Behavior |
|---|---|
| `off` | Native Agent Run only |
| `shadow` | Native executes; LangGraph observes and compares decisions |
| `canary` | Explicit users and deterministic percentage may use LangGraph |
| `active` | Workflow-allowlisted tasks use the primary engine |

All modes support an explicit Native fallback. The environment kill switch
`MEMIND_ORCHESTRATOR_KILL_SWITCH=1` overrides admin configuration and forces
Native selection.

The initial workflow allowlist contains only `code-run-v1`. Ordinary chat and
existing Goosed session traffic remain outside the Orchestrator path.

Portal also has an independent wiring gate:
`MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=1`. It is off by default.
When it is off, Portal does not construct the Shadow observer, query routing
configuration from the Agent Run path, schedule background observations, or
send data to the Orchestrator. Changing memindadm mode alone therefore cannot
put existing Native traffic on the Shadow path.

Page Data validation enforcement has a second independent Portal gate:
`MEMIND_ORCHESTRATOR_PAGE_DATA_VALIDATION_GATE_ENABLED=1`. It is off by
default and becomes effective only while Shadow wiring is effective. When
enabled, a Page Data run remains `running` after the Native delivery guards
finish until the Orchestrator checkpoints a `passed` validation observation.
`failed`, `inconclusive`, not-selected, timeout, and unavailable outcomes fail
closed and prevent the run from being marked successful. The Orchestrator still
does not publish artifacts or receive user content in this mode.

memindadm projects the requested and effective wiring states separately. Canary
readiness requires `shadow_wiring_enabled=true`, so a saved Shadow mode with a
closed Portal gate is visible and cannot be mistaken for a collecting Shadow
deployment.

Phase 2 implemented only the `off` and `shadow` execution semantics. Canary and
Active were initially planning-only so an administrative configuration mistake
could not create two task executors.

Phase 3 makes that restriction explicit in the contract. Routing returns both
`candidateEngine` and `engine`: Canary or Active may nominate LangGraph, while
`engine` remains Native. The decision is projected as
`workflow_execution_planned` with `dryRun=true` and `handoffAllowed=false`.
The execution adapter normalizes authorization, idempotency, timeout,
cancellation, and fallback controls. Phase 5 can transfer ownership only when
the memindadm switch, Portal environment gate, Orchestrator environment gate,
adapter enablement, worker health, authorization, and admission policy all
agree. Any missing gate keeps Native effective.

The Dry-run projection records every code-run decision while the configured
mode is Shadow, Canary, or Active, including `canary_not_selected`. In Shadow,
the same allowlist and deterministic percentage are evaluated in parallel with
the remote observation, so collecting routing evidence does not interrupt
Shadow readiness samples. This prevents a selected-only dataset from reporting
a meaningless 100% candidate rate. The Memind control plane joins each decision
to the Native run terminal state and aggregates candidate rate, selection
reasons, task types, session coverage, and Native terminal coverage without
querying LangGraph.

## Protocol

The framework-neutral contracts are:

- `orchestrator-run-v1`
- `orchestrator-event-v1`
- `workflow-execution-request-v1`
- `workflow-execution-decision-v1`
- `executor-job-request-v1`
- `executor-job-state-v1`
- `executor-dispatch-decision-v1`
- `page-data-validation-observation-v1`

The Phase 3.2 Executor Gateway owns the adapter registry and executor job state
transition boundary. Requests contain workspace/artifact references,
authorization and side-effect policy, timeout, cancellation, fallback, and an
idempotency key. The store interface requires atomic create-if-absent semantics
so retries return the original job and conflicting payloads fail closed.

Goosed, Aider, and OpenHands began as disabled `contract-only` adapters. The
reference in-memory store remains non-durable and is not a deployment backend.
In the current Phase 5 implementation the dispatch capability exists, but
execution and every adapter remain disabled by default. Shadow jobs still record
terminal `blocked` state without launching a process or touching a workspace.

Phase 3.3 persists that state in the Orchestrator-owned PostgreSQL database.
LangGraph checkpoint tables and `executor_jobs` share the isolated
`memind_orchestrator` schema and credentials, but remain behind separate
storage interfaces and readiness probes. This is a deployment simplification,
not a domain ownership leak: neither store can access Memind business tables,
and the Executor Job Store can move to a separate database without changing
the graph or the Memind-facing protocol.

The `build_plan` node creates the deterministic preview job
`<runId>:executor-preview` with idempotency key
`<runId>:build_plan:v1`. The stored result is terminal `blocked` state with
`executor_dispatch_not_implemented`; it is a durable audit and recovery
boundary, not an execution claim. If the run identifier is unsafe or would
overflow the 128-character Job id boundary, the graph substitutes a stable
SHA-256-derived opaque prefix.

Phase 3.4 adds the append-only `executor-job-event-v1` contract. Job snapshots
remain in `executor_jobs`, while cursor-addressable events live in
`executor_job_events`. A transaction commits the initial snapshot and event
together; later state transitions lock the job row and commit the updated
snapshot and next event together. This prevents a visible state transition
without its corresponding audit event.

The read boundary is deliberately narrower than an execution control API:

```text
GET /v1/executor-jobs/:jobId
GET /v1/executor-jobs/:jobId/events?after=<cursor>&limit=<limit>
```

Phase 3.4 had no HTTP submit, retry, claim, or executor cancel endpoint.
Phase 3.5 adds a separately authenticated worker protocol. The control-plane
read projection still contains no task instruction, secret, absolute path, lease
token, or executor SDK object; only a successful worker claim receives the
normalized request and lease token.

The implemented internal API is:

```text
POST /v1/runs
GET /v1/runs/:id
POST /v1/runs/:id/validation-observations
POST /v1/runs/:id/resume
POST /v1/runs/:id/cancel
DELETE /v1/runs/:id
POST /v1/maintenance/purge-terminal-runs
GET /v1/runs/:id/events?after=<cursor>
GET /v1/executor-jobs/:jobId
GET /v1/executor-jobs/:jobId/events?after=<cursor>&limit=<limit>
```

External state changes must use an idempotency key derived from run, graph
version, node, and attempt. Graph state stores resource references and decisions,
not API keys, large logs, binary artifacts, or absolute production paths.

Shadow uses the stricter `control-plane-only-v1` data policy. Portal evaluates
the rollout locally but sends no user message, user identifier, session
identifier, provider key, or workspace path to LangGraph. The remote RunSpec
contains only product run/request identifiers, workflow/task type, configuration
version, and a fixed non-user instruction marker. Terminal runs can be removed
through `DELETE /v1/runs/:id`; deletion removes the LangGraph thread and its
linked terminal Executor Job, whose events cascade in PostgreSQL. Active or
waiting runs fail deletion with `409`.

Page Data code runs may append a post-run
`page-data-validation-observation-v1` after the Native delivery guards finish.
The observation contains only bounded check identifiers, pass/fail/skipped
states, error codes, and aggregate page/publication counts. User prompts, code,
workspace paths, dataset contents, credentials, and database identifiers must
not cross this boundary. The Orchestrator derives and checkpoints a
`passed`/`failed`/`inconclusive` verdict and emits
`workflow_validation_observed`. In the initial observation phase this verdict
does not mutate the Native run result or publication state. With the independent
Page Data validation gate enabled, Portal waits for `passed` before committing
the Native success transition; the default remains observation-only.

The graph keeps three deterministic control-plane nodes:

```text
validate_run -> build_plan -> finalize_run
```

Observe-only runs require `sideEffectsAllowed=false`, project the Native
boundary, and persist a blocked Executor Job. Explicit active runs require
`sideEffectsAllowed=true` plus every execution gate; they queue a job and enter
`waiting` until the worker-owned job reaches a terminal state. LangGraph never
mounts a workspace or imports an executor SDK.

## Phase 5 worker and loop boundary

The task loop is split across durable owners:

```text
LangGraph run: validate -> queue -> wait -> project terminal
Executor Job: queued -> leased -> running -> terminal/retryable
Worker: claim -> heartbeat -> adapter -> artifact references
```

PostgreSQL row locks and lease tokens fence duplicate claims and late worker
updates. Expired leases recover to `retryable` until `maxAttempts` is exhausted,
then become `timed_out`. Worker heartbeats are persisted independently of job
heartbeats so readiness and autoscaling signals also work while the queue is
empty.

Goosed is a remote HTTP/SSE adapter. Aider and OpenHands are process adapters
intended to run in separate worker containers with workspace aliases, an
allowlisted writable root, no shell, bounded environment/output, resource
limits, and no Docker socket. All adapter outputs cross the boundary as bounded
events and artifact references.

Admission is defense in depth: tenant/user/workspace allowlists, global and
per-subject concurrency, and per-minute limits are evaluated before queueing.
The public metrics endpoint exposes queue states, claimable jobs, expired
leases, worker health, and execution-gate state without task contents.

## Failure isolation

Portal invokes Shadow only after the product run and its `queued` event have
been committed. The observation is scheduled without awaiting it. A timeout or
failure:

1. cannot reject or delay `createRun`;
2. cannot mutate the Native run status;
3. is projected as `workflow_shadow_failed`;
4. remains removable by deleting the observer wiring and changing only the
   service URL boundary.

The observer uses a bounded in-process dispatcher. Concurrency defaults to `2`
and the waiting queue to `100`; overflow records `workflow_shadow_skipped` and
never blocks, rejects, or cancels the Native run. Skipped observations are
included in the denominator, shown separately in memindadm, and any non-zero
skip rate blocks Canary readiness.

Successful observations are projected as `workflow_shadow_completed`; graph
checkpoints and blocked Executor Jobs stay in the Orchestrator-owned PostgreSQL
database.

Non-loopback Orchestrator binding requires a service token. Enabling Executor
Jobs requires both distinct service and worker tokens even on loopback.
Non-loopback service URLs are rejected by Portal unless their origin is
declared through `MEMIND_ORCHESTRATOR_URL` or
`MEMIND_ORCHESTRATOR_ALLOWED_ORIGINS`; loopback origins remain allowed for
local validation.

Terminal retention is independently disabled by default. Setting
`MEMIND_ORCHESTRATOR_RETENTION_DAYS` to a positive value starts a bounded
periodic sweep after service startup; it uses the same terminal-only deletion
path and never deletes waiting/running workflows. The authenticated maintenance
endpoint defaults to dry-run and requires `apply=true` for mutation.

Both terminal projection events contain bounded observation latency. The
Memind-owned observability service aggregates those events and may join them to
the product run status. Only per-run detail calls cross the service boundary to
read LangGraph checkpoint state and node events.

Canary readiness is also a Memind control-plane projection. It excludes
synthetic smoke runs and evaluates operational health, durable checkpoint state,
durable Executor Job storage, sample volume, session coverage, success rate,
latency coverage, P95 latency, Native terminal coverage, and sample freshness.
It also requires the independent Portal Shadow wiring gate to be effective.
The result is advisory:
`manual_canary_review` never mutates routing configuration or transfers
execution ownership.

## Deployment evolution

1. Local native Orchestrator process with an explicit MemorySaver for debugging.
2. Colima Compose with an isolated PostgreSQL checkpoint database.
3. Approved server-side shadow service; no production task claim.
4. User-allowlisted code-run canary with Native rollback.
5. Isolated Aider/OpenHands executor workers.
6. Move the same service contract to Linux or Kubernetes when horizontal scaling
   or independent ownership becomes necessary.

The Orchestrator must not share the Goosed Compose lifecycle or mount the Docker
socket. It calls MindSpace through its service API and calls Goosed through the
existing proxy boundary.