memind/docs/architecture/memind-orchestrator-boundary.md

Memind Workflow Orchestrator Boundary

Decision

The Orchestrator starts in the Memind repository but is designed as an independently deployable bounded context. Source colocation does not authorize in-process execution inside Portal.

The stable boundary is:

Memind Control Plane
  -> versioned Orchestrator API / events
Workflow Orchestrator
  -> versioned Executor Gateway
Executor adapters
  -> Goosed / Aider / OpenHands

LangGraph remains an internal workflow engine implementation. Memind must not depend on LangGraph graph, checkpoint, command, or interrupt types.

Ownership

Data	Owner
User, authorization, billing, product run	Memind
Workflow node state, checkpoint, interrupt	Orchestrator
Goosed session	Goosed plus Session Broker mapping
Executor job	Executor Gateway / executor adapter
Workspace and deliverable	Workspace / Artifact / MindSpace services
User-visible progress and audit	Memind run-event projection

The Orchestrator must not read or write Memind user, billing, capability, or provider-key tables. A shared PostgreSQL instance is acceptable during the single-host phase, but the Orchestrator must use its own database and database credentials.

Runtime modes

memindadm owns the versioned routing configuration:

Mode	Behavior
off	Native Agent Run only
shadow	Native executes; LangGraph observes and compares decisions
canary	Explicit users and deterministic percentage may use LangGraph
active	Workflow-allowlisted tasks use the primary engine

All modes support an explicit Native fallback. The environment kill switch MEMIND_ORCHESTRATOR_KILL_SWITCH=1 overrides admin configuration and forces Native selection.

The initial workflow allowlist contains only code-run-v1. Ordinary chat and existing Goosed session traffic remain outside the Orchestrator path.

Phase 2 implemented only the off and shadow execution semantics. Canary and Active were initially planning-only so an administrative configuration mistake could not create two task executors.

Phase 3 makes that restriction explicit in the contract. Routing returns both candidateEngine and engine: Canary or Active may nominate LangGraph, while engine remains Native. The decision is projected as workflow_execution_planned with dryRun=true and handoffAllowed=false. The execution adapter normalizes authorization, idempotency, timeout, cancellation, and fallback controls. Phase 5 can transfer ownership only when the memindadm switch, Portal environment gate, Orchestrator environment gate, adapter enablement, worker health, authorization, and admission policy all agree. Any missing gate keeps Native effective.

The Dry-run projection records every code-run decision while the configured mode is Shadow, Canary, or Active, including canary_not_selected. In Shadow, the same allowlist and deterministic percentage are evaluated in parallel with the remote observation, so collecting routing evidence does not interrupt Shadow readiness samples. This prevents a selected-only dataset from reporting a meaningless 100% candidate rate. The Memind control plane joins each decision to the Native run terminal state and aggregates candidate rate, selection reasons, task types, session coverage, and Native terminal coverage without querying LangGraph.

Protocol

The framework-neutral contracts are:

• orchestrator-run-v1
• orchestrator-event-v1
• workflow-execution-request-v1
• workflow-execution-decision-v1
• executor-job-request-v1
• executor-job-state-v1
• executor-dispatch-decision-v1

The Phase 3.2 Executor Gateway owns the adapter registry and executor job state transition boundary. Requests contain workspace/artifact references, authorization and side-effect policy, timeout, cancellation, fallback, and an idempotency key. The store interface requires atomic create-if-absent semantics so retries return the original job and conflicting payloads fail closed.

Goosed, Aider, and OpenHands began as disabled contract-only adapters. The reference in-memory store remains non-durable and is not a deployment backend. In the current Phase 5 implementation the dispatch capability exists, but execution and every adapter remain disabled by default. Shadow jobs still record terminal blocked state without launching a process or touching a workspace.

Phase 3.3 persists that state in the Orchestrator-owned PostgreSQL database. LangGraph checkpoint tables and executor_jobs share the isolated memind_orchestrator schema and credentials, but remain behind separate storage interfaces and readiness probes. This is a deployment simplification, not a domain ownership leak: neither store can access Memind business tables, and the Executor Job Store can move to a separate database without changing the graph or the Memind-facing protocol.

The build_plan node creates the deterministic preview job <runId>:executor-preview with idempotency key <runId>:build_plan:v1. The stored result is terminal blocked state with executor_dispatch_not_implemented; it is a durable audit and recovery boundary, not an execution claim. If the run identifier is unsafe or would overflow the 128-character Job id boundary, the graph substitutes a stable SHA-256-derived opaque prefix.

Phase 3.4 adds the append-only executor-job-event-v1 contract. Job snapshots remain in executor_jobs, while cursor-addressable events live in executor_job_events. A transaction commits the initial snapshot and event together; later state transitions lock the job row and commit the updated snapshot and next event together. This prevents a visible state transition without its corresponding audit event.

The read boundary is deliberately narrower than an execution control API:

GET /v1/executor-jobs/:jobId
GET /v1/executor-jobs/:jobId/events?after=<cursor>&limit=<limit>

Phase 3.4 had no HTTP submit, retry, claim, or executor cancel endpoint. Phase 3.5 adds a separately authenticated worker protocol. The control-plane read projection still contains no task instruction, secret, absolute path, lease token, or executor SDK object; only a successful worker claim receives the normalized request and lease token.

The implemented internal API is:

POST /v1/runs
GET /v1/runs/:id
POST /v1/runs/:id/resume
POST /v1/runs/:id/cancel
GET /v1/runs/:id/events?after=<cursor>
GET /v1/executor-jobs/:jobId
GET /v1/executor-jobs/:jobId/events?after=<cursor>&limit=<limit>

External state changes must use an idempotency key derived from run, graph version, node, and attempt. Graph state stores resource references and decisions, not API keys, large logs, binary artifacts, or absolute production paths.

The graph keeps three deterministic control-plane nodes:

validate_run -> build_plan -> finalize_run

Observe-only runs require sideEffectsAllowed=false, project the Native boundary, and persist a blocked Executor Job. Explicit active runs require sideEffectsAllowed=true plus every execution gate; they queue a job and enter waiting until the worker-owned job reaches a terminal state. LangGraph never mounts a workspace or imports an executor SDK.

Phase 5 worker and loop boundary

The task loop is split across durable owners:

LangGraph run: validate -> queue -> wait -> project terminal
Executor Job: queued -> leased -> running -> terminal/retryable
Worker: claim -> heartbeat -> adapter -> artifact references

PostgreSQL row locks and lease tokens fence duplicate claims and late worker updates. Expired leases recover to retryable until maxAttempts is exhausted, then become timed_out. Worker heartbeats are persisted independently of job heartbeats so readiness and autoscaling signals also work while the queue is empty.

Goosed is a remote HTTP/SSE adapter. Aider and OpenHands are process adapters intended to run in separate worker containers with workspace aliases, an allowlisted writable root, no shell, bounded environment/output, resource limits, and no Docker socket. All adapter outputs cross the boundary as bounded events and artifact references.

Admission is defense in depth: tenant/user/workspace allowlists, global and per-subject concurrency, and per-minute limits are evaluated before queueing. The public metrics endpoint exposes queue states, claimable jobs, expired leases, worker health, and execution-gate state without task contents.

Failure isolation

Portal invokes Shadow only after the product run and its queued event have been committed. The observation is scheduled without awaiting it. A timeout or failure:

1. cannot reject or delay createRun;
2. cannot mutate the Native run status;
3. is projected as workflow_shadow_failed;
4. remains removable by deleting the observer wiring and changing only the service URL boundary.

Successful observations are projected as workflow_shadow_completed; graph checkpoints and blocked Executor Jobs stay in the Orchestrator-owned PostgreSQL database.

Both terminal projection events contain bounded observation latency. The Memind-owned observability service aggregates those events and may join them to the product run status. Only per-run detail calls cross the service boundary to read LangGraph checkpoint state and node events.

Canary readiness is also a Memind control-plane projection. It excludes synthetic smoke runs and evaluates operational health, durable checkpoint state, durable Executor Job storage, sample volume, session coverage, success rate, latency coverage, P95 latency, Native terminal coverage, and sample freshness. The result is advisory: manual_canary_review never mutates routing configuration or transfers execution ownership.

Deployment evolution

1. Local native Orchestrator process with an explicit MemorySaver for debugging.
2. Colima Compose with an isolated PostgreSQL checkpoint database.
3. Approved server-side shadow service; no production task claim.
4. User-allowlisted code-run canary with Native rollback.
5. Isolated Aider/OpenHands executor workers.
6. Move the same service contract to Linux or Kubernetes when horizontal scaling or independent ownership becomes necessary.

The Orchestrator must not share the Goosed Compose lifecycle or mount the Docker socket. It calls MindSpace through its service API and calls Goosed through the existing proxy boundary.