memind/docs/architecture/memind-orchestrator-boundary.md

Memind Workflow Orchestrator Boundary

Decision

The Orchestrator starts in the Memind repository but is designed as an independently deployable bounded context. Source colocation does not authorize in-process execution inside Portal.

The stable boundary is:

Memind Control Plane
  -> versioned Orchestrator API / events
Workflow Orchestrator
  -> versioned Executor Gateway
Executor adapters
  -> Goosed / Aider / OpenHands

LangGraph remains an internal workflow engine implementation. Memind must not depend on LangGraph graph, checkpoint, command, or interrupt types.

Ownership

Data	Owner
User, authorization, billing, product run	Memind
Workflow node state, checkpoint, interrupt	Orchestrator
Goosed session	Goosed plus Session Broker mapping
Executor job	Executor Gateway / executor adapter
Workspace and deliverable	Workspace / Artifact / MindSpace services
User-visible progress and audit	Memind run-event projection

The Orchestrator must not read or write Memind user, billing, capability, or provider-key tables. A shared PostgreSQL instance is acceptable during the single-host phase, but the Orchestrator must use its own database and database credentials.

Runtime modes

memindadm owns the versioned routing configuration:

Mode	Behavior
off	Native Agent Run only
shadow	Native executes; LangGraph observes and compares decisions
canary	Explicit users and deterministic percentage may use LangGraph
active	Workflow-allowlisted tasks use the primary engine

All modes support an explicit Native fallback. The environment kill switch MEMIND_ORCHESTRATOR_KILL_SWITCH=1 overrides admin configuration and forces Native selection.

The initial workflow allowlist contains only code-run-v1. Ordinary chat and existing Goosed session traffic remain outside the Orchestrator path.

Portal also has an independent wiring gate: MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=1. It is off by default. When it is off, Portal does not construct the Shadow observer, query routing configuration from the Agent Run path, schedule background observations, or send data to the Orchestrator. Changing memindadm mode alone therefore cannot put existing Native traffic on the Shadow path.

Page Data validation enforcement has a second independent Portal gate: MEMIND_ORCHESTRATOR_PAGE_DATA_VALIDATION_GATE_ENABLED=1. It is off by default and becomes effective only while Shadow wiring is effective. When enabled, a Page Data run remains running after the Native delivery guards finish until the Orchestrator checkpoints a passed validation observation. failed, inconclusive, not-selected, timeout, and unavailable outcomes fail closed and prevent the run from being marked successful. The Orchestrator still does not publish artifacts or receive user content in this mode.

memindadm projects the requested and effective wiring states separately. Canary readiness requires shadow_wiring_enabled=true, so a saved Shadow mode with a closed Portal gate is visible and cannot be mistaken for a collecting Shadow deployment.

Phase 2 implemented only the off and shadow execution semantics. Canary and Active were initially planning-only so an administrative configuration mistake could not create two task executors.

Phase 3 makes that restriction explicit in the contract. Routing returns both candidateEngine and engine: Canary or Active may nominate LangGraph, while engine remains Native. The decision is projected as workflow_execution_planned with dryRun=true and handoffAllowed=false. The execution adapter normalizes authorization, idempotency, timeout, cancellation, and fallback controls. Phase 5 can transfer ownership only when the memindadm switch, Portal environment gate, Orchestrator environment gate, adapter enablement, worker health, authorization, and admission policy all agree. Any missing gate keeps Native effective.

The Dry-run projection records every code-run decision while the configured mode is Shadow, Canary, or Active, including canary_not_selected. In Shadow, the same allowlist and deterministic percentage are evaluated in parallel with the remote observation, so collecting routing evidence does not interrupt Shadow readiness samples. This prevents a selected-only dataset from reporting a meaningless 100% candidate rate. The Memind control plane joins each decision to the Native run terminal state and aggregates candidate rate, selection reasons, task types, session coverage, and Native terminal coverage without querying LangGraph.

Protocol

The framework-neutral contracts are:

• orchestrator-run-v1
• orchestrator-event-v1
• workflow-execution-request-v1
• workflow-execution-decision-v1
• executor-job-request-v1
• executor-job-state-v1
• executor-dispatch-decision-v1
• page-data-validation-observation-v1

The Phase 3.2 Executor Gateway owns the adapter registry and executor job state transition boundary. Requests contain workspace/artifact references, authorization and side-effect policy, timeout, cancellation, fallback, and an idempotency key. The store interface requires atomic create-if-absent semantics so retries return the original job and conflicting payloads fail closed.

Goosed, Aider, and OpenHands began as disabled contract-only adapters. The reference in-memory store remains non-durable and is not a deployment backend. In the current Phase 5 implementation the dispatch capability exists, but execution and every adapter remain disabled by default. Shadow jobs still record terminal blocked state without launching a process or touching a workspace.

Phase 3.3 persists that state in the Orchestrator-owned PostgreSQL database. LangGraph checkpoint tables and executor_jobs share the isolated memind_orchestrator schema and credentials, but remain behind separate storage interfaces and readiness probes. This is a deployment simplification, not a domain ownership leak: neither store can access Memind business tables, and the Executor Job Store can move to a separate database without changing the graph or the Memind-facing protocol.

The build_plan node creates the deterministic preview job <runId>:executor-preview with idempotency key <runId>:build_plan:v1. The stored result is terminal blocked state with executor_dispatch_not_implemented; it is a durable audit and recovery boundary, not an execution claim. If the run identifier is unsafe or would overflow the 128-character Job id boundary, the graph substitutes a stable SHA-256-derived opaque prefix.

Phase 3.4 adds the append-only executor-job-event-v1 contract. Job snapshots remain in executor_jobs, while cursor-addressable events live in executor_job_events. A transaction commits the initial snapshot and event together; later state transitions lock the job row and commit the updated snapshot and next event together. This prevents a visible state transition without its corresponding audit event.

The read boundary is deliberately narrower than an execution control API:

GET /v1/executor-jobs/:jobId
GET /v1/executor-jobs/:jobId/events?after=<cursor>&limit=<limit>

Phase 3.4 had no HTTP submit, retry, claim, or executor cancel endpoint. Phase 3.5 adds a separately authenticated worker protocol. The control-plane read projection still contains no task instruction, secret, absolute path, lease token, or executor SDK object; only a successful worker claim receives the normalized request and lease token.

The implemented internal API is:

POST /v1/runs
GET /v1/runs/:id
POST /v1/runs/:id/validation-observations
POST /v1/runs/:id/resume
POST /v1/runs/:id/cancel
DELETE /v1/runs/:id
POST /v1/maintenance/purge-terminal-runs
GET /v1/runs/:id/events?after=<cursor>
GET /v1/executor-jobs/:jobId
GET /v1/executor-jobs/:jobId/events?after=<cursor>&limit=<limit>

External state changes must use an idempotency key derived from run, graph version, node, and attempt. Graph state stores resource references and decisions, not API keys, large logs, binary artifacts, or absolute production paths.

Shadow uses the stricter control-plane-only-v1 data policy. Portal evaluates the rollout locally but sends no user message, user identifier, session identifier, provider key, or workspace path to LangGraph. The remote RunSpec contains only product run/request identifiers, workflow/task type, configuration version, and a fixed non-user instruction marker. Terminal runs can be removed through DELETE /v1/runs/:id; deletion removes the LangGraph thread and its linked terminal Executor Job, whose events cascade in PostgreSQL. Active or waiting runs fail deletion with 409.

Page Data code runs may append a post-run page-data-validation-observation-v1 after the Native delivery guards finish. The observation contains only bounded check identifiers, pass/fail/skipped states, error codes, and aggregate page/publication counts. User prompts, code, workspace paths, dataset contents, credentials, and database identifiers must not cross this boundary. The Orchestrator derives and checkpoints a passed/failed/inconclusive verdict and emits workflow_validation_observed. In the initial observation phase this verdict does not mutate the Native run result or publication state. With the independent Page Data validation gate enabled, Portal waits for passed before committing the Native success transition; the default remains observation-only.

The graph keeps three deterministic control-plane nodes:

validate_run -> build_plan -> finalize_run

Observe-only runs require sideEffectsAllowed=false, project the Native boundary, and persist a blocked Executor Job. Explicit active runs require sideEffectsAllowed=true plus every execution gate; they queue a job and enter waiting until the worker-owned job reaches a terminal state. LangGraph never mounts a workspace or imports an executor SDK.

Phase 5 worker and loop boundary

The task loop is split across durable owners:

LangGraph run: validate -> queue -> wait -> project terminal
Executor Job: queued -> leased -> running -> terminal/retryable
Worker: claim -> heartbeat -> adapter -> artifact references

PostgreSQL row locks and lease tokens fence duplicate claims and late worker updates. Expired leases recover to retryable until maxAttempts is exhausted, then become timed_out. Worker heartbeats are persisted independently of job heartbeats so readiness and autoscaling signals also work while the queue is empty.

Goosed is a remote HTTP/SSE adapter. Aider and OpenHands are process adapters intended to run in separate worker containers with workspace aliases, an allowlisted writable root, no shell, bounded environment/output, resource limits, and no Docker socket. All adapter outputs cross the boundary as bounded events and artifact references.

Admission is defense in depth: tenant/user/workspace allowlists, global and per-subject concurrency, and per-minute limits are evaluated before queueing. The public metrics endpoint exposes queue states, claimable jobs, expired leases, worker health, and execution-gate state without task contents.

Failure isolation

Portal invokes Shadow only after the product run and its queued event have been committed. The observation is scheduled without awaiting it. A timeout or failure:

1. cannot reject or delay createRun;
2. cannot mutate the Native run status;
3. is projected as workflow_shadow_failed;
4. remains removable by deleting the observer wiring and changing only the service URL boundary.

The observer uses a bounded in-process dispatcher. Concurrency defaults to 2 and the waiting queue to 100; overflow records workflow_shadow_skipped and never blocks, rejects, or cancels the Native run. Skipped observations are included in the denominator, shown separately in memindadm, and any non-zero skip rate blocks Canary readiness.

Successful observations are projected as workflow_shadow_completed; graph checkpoints and blocked Executor Jobs stay in the Orchestrator-owned PostgreSQL database.

Non-loopback Orchestrator binding requires a service token. Enabling Executor Jobs requires both distinct service and worker tokens even on loopback. Non-loopback service URLs are rejected by Portal unless their origin is declared through MEMIND_ORCHESTRATOR_URL or MEMIND_ORCHESTRATOR_ALLOWED_ORIGINS; loopback origins remain allowed for local validation.

Terminal retention is independently disabled by default. Setting MEMIND_ORCHESTRATOR_RETENTION_DAYS to a positive value starts a bounded periodic sweep after service startup; it uses the same terminal-only deletion path and never deletes waiting/running workflows. The authenticated maintenance endpoint defaults to dry-run and requires apply=true for mutation.

Both terminal projection events contain bounded observation latency. The Memind-owned observability service aggregates those events and may join them to the product run status. Only per-run detail calls cross the service boundary to read LangGraph checkpoint state and node events.

Canary readiness is also a Memind control-plane projection. It excludes synthetic smoke runs and evaluates operational health, durable checkpoint state, durable Executor Job storage, sample volume, session coverage, success rate, latency coverage, P95 latency, Native terminal coverage, and sample freshness. It also requires the independent Portal Shadow wiring gate to be effective. The result is advisory: manual_canary_review never mutates routing configuration or transfers execution ownership.

Deployment evolution

1. Local native Orchestrator process with an explicit MemorySaver for debugging.
2. Colima Compose with an isolated PostgreSQL checkpoint database.
3. Approved server-side shadow service; no production task claim.
4. User-allowlisted code-run canary with Native rollback.
5. Isolated Aider/OpenHands executor workers.
6. Move the same service contract to Linux or Kubernetes when horizontal scaling or independent ownership becomes necessary.

The Orchestrator must not share the Goosed Compose lifecycle or mount the Docker socket. It calls MindSpace through its service API and calls Goosed through the existing proxy boundary.