memind/services/orchestrator

Memind Workflow Orchestrator

This directory is the extraction boundary for Memind's durable workflow runtime. It is intentionally source-colocated with Memind while remaining framework-neutral at its public edge.

Dependency rule

Allowed:

Memind -> orchestrator contracts / client
Orchestrator -> contracts / executor adapters
Executor adapters -> versioned internal APIs

Forbidden:

Orchestrator -> server.mjs
Orchestrator -> user-auth.mjs
Orchestrator -> Memind business tables
Orchestrator -> production filesystem paths
Memind -> LangGraph StateGraph / Command / checkpoint types

LangGraph is an implementation of the generic workflow engine contract. The Memind-facing protocol is orchestrator-run-v1 plus orchestrator-event-v1.

Current stage

Phase 1 established:

• framework-neutral run and event contracts;
• a plug-in workflow engine registry;
• a remote workflow engine client;
• versioned admin configuration;
• deterministic Off / Shadow / Canary / Active routing decisions;
• an environment-level emergency kill switch.

Phase 2 adds:

• a real LangGraph StateGraph service on the versioned HTTP boundary;
• a PostgreSQL PostgresSaver checkpoint owner;
• an explicitly non-durable MemorySaver option for tests and local debugging;
• a code-run-v1 observe-only graph with no executor dispatch or tool access;
• a Portal fire-and-forget shadow projection;
• live health probing from memindadm;
• a separate Colima/Compose deployment artifact.

Phase 2.5 adds a Memind-owned Shadow observability projection:

• success/failure totals and rates;
• P50/P95 observation latency;
• Native terminal status beside the Shadow result;
• recent failure codes and messages;
• per-run LangGraph checkpoint and node-event inspection through the versioned remote API.

The projection reads h5_agent_run_events from the Memind control plane. LangGraph still has no access to Memind business tables.

Phase 2.6 adds a read-only Canary readiness gate. It evaluates a fixed seven-day window and never changes the runtime mode. The first gate requires:

• at least 20 non-smoke observations across at least five real sessions;
• at least 95% Shadow success, latency coverage, and Native terminal coverage;
• Shadow P95 latency no greater than two seconds;
• a healthy observe-only service backed by a durable checkpoint;
• a fresh, uncapped metric window while the control plane remains in Shadow.

Passing the gate means only manual_canary_review; it does not authorize or activate Canary routing.

Phase 3 introduced the execution handoff boundary in a hard-disabled state:

• workflow-execution-request-v1 normalizes authorization, timeout, cancellation, fallback, and idempotency controls;
• workflow-execution-decision-v1 separates the candidate engine from the effective executor;
• Canary and Active selections emit auditable workflow_execution_planned events;
• the effective engine remains Native and non-Native dispatch throws WORKFLOW_EXECUTION_HANDOFF_DISABLED;
• the implementation capability was hard-disabled in code while the contracts and observability matured.

Phase 3.1 adds a Memind-owned Dry-run projection. Shadow mode continues its LangGraph observation while also evaluating the configured Canary allowlist and percentage. It records both selected and non-selected decisions so the candidate selection rate has a complete denominator. memindadm shows candidate reasons, task-type distribution, Native terminal coverage, recent decisions, and an explicit alert if any stored decision ever reports handoffAllowed=true.

Phase 3.2 establishes the Executor Gateway boundary:

• executor-job-request-v1, executor-job-state-v1, and executor-dispatch-decision-v1 remain independent of Goosed, Aider, OpenHands, and LangGraph SDK types;
• jobs use workspace and artifact references rather than host filesystem paths;
• the store contract provides atomic idempotency, lookup, and state update operations;
• timeout, cancellation, fallback, authorization, network, and side-effect controls are normalized before adapter selection;
• Goosed, Aider, and OpenHands were initially registered as contract-only adapters;
• the reference memory store is explicitly non-durable and used only for local contract tests.

Phase 3.2 could record blocked, idempotent job state but could not invoke an adapter or launch a process.

Phase 3.3 makes that blocked state durable:

• the Orchestrator owns a PostgreSQL executor_jobs table beside its LangGraph checkpoint tables, under the same isolated schema and database credentials;
• the build_plan node atomically creates one deterministic Executor Job per run and graph-node version;
• retries recover the same job through its idempotency key, while a changed semantic request fails closed;
• /health and /ready report and probe checkpoint and Executor Job storage independently;
• memory mode keeps both stores explicitly non-durable.

The Phase 3.3 persisted job was a preview terminal state. Existing Shadow runs remain blocked and non-executable after later phases.

Phase 3.4 adds read-only Executor Job observability:

• executor-job-event-v1 is stored in the separate PostgreSQL executor_job_events table with a per-job sequence cursor;
• initial state and its first event are committed atomically, as are later state transitions and their events;
• existing Phase 3.3 snapshots receive a deterministic executor_job_snapshot_imported migration event;
• the versioned API exposes job state and cursor-paged events without exposing task instructions, credentials, workspace paths, or executor SDK types;
• unsafe or oversized run identifiers are represented by a deterministic SHA-256-derived opaque Executor Job id so PostgreSQL and URL identifiers stay bounded;
• Shadow Run detail in memindadm joins Native state, LangGraph checkpoints, Executor Job state, and Executor Job events;
• Canary readiness now requires both checkpoint and Executor Job storage to be durable.

Phase 3.5 through Phase 5 add the opt-in execution plane:

• executor-worker-v1 provides atomic claim, lease fencing, heartbeat, progress, retry, timeout recovery, completion, cancellation, and drain semantics;
• PostgreSQL stores worker heartbeats and exposes queue/worker metrics;
• Goosed uses the existing session HTTP/SSE transport with abort propagation;
• Aider and OpenHands run as separate worker processes or containers with workspace aliases, isolated roots, no shell, bounded environment/output, and artifact references;
• LangGraph active runs queue an Executor Job, wait durably, and converge their terminal state from the job/event boundary;
• admission policy enforces tenant/user/workspace allowlists, concurrency, and per-minute limits before a job can enter the queue;
• memindadm, Portal environment, Orchestrator environment, adapter enablement, worker health, authorization, and admission are independent fail-closed gates.

The default memindadm switch and both environment execution gates remain off. Native Agent Run therefore remains the executor until an explicitly approved Canary satisfies every gate.

Admin configuration

The memindadm page is /ops/admin/orchestrator.

Supported modes:

• off: Native Agent Run only.
• shadow: Native executes; the configured engine may observe.
• canary: user allowlist and deterministic rollout percentage select eligible runs; execution still requires the explicit execution switch and environment gates.
• active: workflow-allowlisted tasks select the primary engine, subject to the same health, policy, worker, and kill-switch gates.

MEMIND_ORCHESTRATOR_KILL_SWITCH=1 always forces Native selection.

Local process

PostgreSQL is the default and required checkpoint and Executor Job backend:

MEMIND_ORCHESTRATOR_DATABASE_URL='postgresql://...' \
MEMIND_ORCHESTRATOR_SERVICE_TOKEN='...' \
pnpm dev:orchestrator

Memory mode is intentionally opt-in and non-durable:

MEMIND_ORCHESTRATOR_CHECKPOINT_MODE=memory pnpm dev:orchestrator

The service binds to 127.0.0.1:8093 by default. Configure the same URL and service token in Portal, set MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED=1 on that Portal instance, then enable shadow in /ops/admin/orchestrator. Without the Portal gate, Native code runs do not construct or call the observer.

The admin runtime projection distinguishes requested Shadow mode from effective Portal wiring. Canary readiness requires shadow_wiring_enabled=true; a closed environment gate therefore blocks promotion even when older successful samples are still present.

memindadm exposes the projection through:

GET /admin-api/orchestrator/shadow-runs
GET /admin-api/orchestrator/shadow-runs/:runId
GET /admin-api/orchestrator/canary-readiness
GET /admin-api/orchestrator/execution-plans

Selecting a Shadow Run in memindadm also reads the linked Executor Job through the Orchestrator API:

GET /v1/executor-jobs/:jobId
GET /v1/executor-jobs/:jobId/events?after=<cursor>&limit=<limit>
POST /v1/executor-jobs
POST /v1/workers/claim
POST /v1/workers/jobs/:jobId/start
POST /v1/workers/jobs/:jobId/heartbeat
POST /v1/workers/jobs/:jobId/progress
POST /v1/workers/jobs/:jobId/complete
POST /v1/workers/jobs/:jobId/fail
POST /v1/workers/recover-expired
GET /v1/workers/stats
DELETE /v1/runs/:runId
POST /v1/maintenance/purge-terminal-runs

Shadow RunSpecs use the control-plane-only-v1 policy and do not copy user messages, user IDs, or session IDs into Orchestrator storage. Deleting a terminal run removes its checkpoint thread and linked terminal Executor Job.

Colima deployment

Colima is the recommended first container host on macOS because this service and its PostgreSQL checkpoint database have an isolated Compose lifecycle:

cd deploy/orchestrator
cp .env.example .env
# Replace both example secrets before starting.
docker compose up -d --build
docker compose ps
curl http://127.0.0.1:8093/ready

When Portal or the Shadow smoke process runs outside Compose, export the same service token into the child process environment. Plain source does not export new shell variables by itself:

set -a
source deploy/orchestrator/.env
set +a
export MEMIND_ORCHESTRATOR_URL=http://127.0.0.1:8093
pnpm dev

The Compose project:

• publishes only the Orchestrator port on loopback;
• does not publish PostgreSQL;
• does not mount the Docker socket or a Memind workspace;
• keeps PostgreSQL on a dedicated internal network while the service also joins a separate edge network for its loopback-published port;
• is not part of the Goosed Compose lifecycle.

This local Compose artifact is not a production release path. Production still requires the repository release gates and a separately approved deployment.

Runtime environment

Variable	Purpose	Default
MEMIND_ORCHESTRATOR_HOST	HTTP bind address	127.0.0.1
MEMIND_ORCHESTRATOR_PORT	HTTP port	8093
MEMIND_ORCHESTRATOR_SERVICE_TOKEN	Bearer token for /v1/*	empty
MEMIND_ORCHESTRATOR_WORKER_TOKEN	Additional worker-protocol token	empty
MEMIND_ORCHESTRATOR_EXECUTION_ENABLED	Queue admission execution gate	false
MEMIND_ORCHESTRATOR_ENABLED_EXECUTORS	Enabled adapter IDs	empty
MEMIND_ORCHESTRATOR_*_ALLOWLIST	Tenant/user/workspace admission scopes	empty / workspace alias
MEMIND_ORCHESTRATOR_MAX_*	Global, tenant, user and rate quotas	bounded safe defaults
MEMIND_ORCHESTRATOR_CHECKPOINT_MODE	postgres or explicit memory	postgres
MEMIND_ORCHESTRATOR_DATABASE_URL	Dedicated Orchestrator PostgreSQL URL	required
MEMIND_ORCHESTRATOR_DATABASE_SCHEMA	Checkpoint and Executor Job schema	memind_orchestrator
MEMIND_ORCHESTRATOR_RETENTION_DAYS	Delete terminal workflow data older than this age; 0 disables	0
MEMIND_ORCHESTRATOR_RETENTION_SWEEP_INTERVAL_MS	Retention sweep interval, minimum one hour	86400000
MEMIND_ORCHESTRATOR_RETENTION_SWEEP_LIMIT	Maximum terminal candidates per sweep	100

Portal-side variables:

Variable	Purpose	Default
MEMIND_ORCHESTRATOR_SHADOW_OBSERVATION_ENABLED	Construct and schedule the Shadow observer	false
MEMIND_ORCHESTRATOR_SHADOW_MAX_CONCURRENCY	Maximum simultaneous observations	2
MEMIND_ORCHESTRATOR_SHADOW_MAX_QUEUE	Maximum waiting observations before skip	100
MEMIND_ORCHESTRATOR_ALLOWED_ORIGINS	Additional exact non-loopback Orchestrator origins	empty

An Orchestrator bound beyond loopback refuses to start without the service token. Execution refuses to start without both service and worker tokens. Retention is disabled by default; the maintenance endpoint is dry-run unless the authenticated caller sends apply=true.

Extraction test

The service is ready to move into a separate repository only when:

1. it can run against a mock Memind control plane;
2. it owns its checkpoint database and credentials;
3. task inputs use resource references rather than host paths;
4. executor calls use versioned adapters;
5. moving the service requires only changing ORCHESTRATOR_URL.