memind/services/orchestrator

Memind Workflow Orchestrator

This directory is the extraction boundary for Memind's durable workflow runtime. It is intentionally source-colocated with Memind while remaining framework-neutral at its public edge.

Dependency rule

Allowed:

Memind -> orchestrator contracts / client
Orchestrator -> contracts / executor adapters
Executor adapters -> versioned internal APIs

Forbidden:

Orchestrator -> server.mjs
Orchestrator -> user-auth.mjs
Orchestrator -> Memind business tables
Orchestrator -> production filesystem paths
Memind -> LangGraph StateGraph / Command / checkpoint types

LangGraph is an implementation of the generic workflow engine contract. The Memind-facing protocol is orchestrator-run-v1 plus orchestrator-event-v1.

Current stage

Phase 1 established:

• framework-neutral run and event contracts;
• a plug-in workflow engine registry;
• a remote workflow engine client;
• versioned admin configuration;
• deterministic Off / Shadow / Canary / Active routing decisions;
• an environment-level emergency kill switch.

Phase 2 adds:

• a real LangGraph StateGraph service on the versioned HTTP boundary;
• a PostgreSQL PostgresSaver checkpoint owner;
• an explicitly non-durable MemorySaver option for tests and local debugging;
• a code-run-v1 observe-only graph with no executor dispatch or tool access;
• a Portal fire-and-forget shadow projection;
• live health probing from memindadm;
• a separate Colima/Compose deployment artifact.

Phase 2.5 adds a Memind-owned Shadow observability projection:

• success/failure totals and rates;
• P50/P95 observation latency;
• Native terminal status beside the Shadow result;
• recent failure codes and messages;
• per-run LangGraph checkpoint and node-event inspection through the versioned remote API.

The projection reads h5_agent_run_events from the Memind control plane. LangGraph still has no access to Memind business tables.

Phase 2.6 adds a read-only Canary readiness gate. It evaluates a fixed seven-day window and never changes the runtime mode. The first gate requires:

• at least 20 non-smoke observations across at least five real sessions;
• at least 95% Shadow success, latency coverage, and Native terminal coverage;
• Shadow P95 latency no greater than two seconds;
• a healthy observe-only service backed by a durable checkpoint;
• a fresh, uncapped metric window while the control plane remains in Shadow.

Passing the gate means only manual_canary_review; it does not authorize or activate Canary routing.

Phase 3 adds the execution handoff boundary without enabling it:

• workflow-execution-request-v1 normalizes authorization, timeout, cancellation, fallback, and idempotency controls;
• workflow-execution-decision-v1 separates the candidate engine from the effective executor;
• Canary and Active selections emit auditable workflow_execution_planned events;
• the effective engine remains Native and non-Native dispatch throws WORKFLOW_EXECUTION_HANDOFF_DISABLED;
• the implementation capability is hard-disabled in code, so memindadm configuration cannot accidentally unlock execution.

Phase 3.1 adds a Memind-owned Dry-run projection. Shadow mode continues its LangGraph observation while also evaluating the configured Canary allowlist and percentage. It records both selected and non-selected decisions so the candidate selection rate has a complete denominator. memindadm shows candidate reasons, task-type distribution, Native terminal coverage, recent decisions, and an explicit alert if any stored decision ever reports handoffAllowed=true.

Phase 3.2 establishes the Executor Gateway boundary:

• executor-job-request-v1, executor-job-state-v1, and executor-dispatch-decision-v1 remain independent of Goosed, Aider, OpenHands, and LangGraph SDK types;
• jobs use workspace and artifact references rather than host filesystem paths;
• the store contract provides atomic idempotency, lookup, and state update operations;
• timeout, cancellation, fallback, authorization, network, and side-effect controls are normalized before adapter selection;
• Goosed, Aider, and OpenHands are registered as contract-only adapters;
• the reference memory store is explicitly non-durable and used only for local contract tests.

EXECUTOR_DISPATCH_IMPLEMENTED=false is a code-level gate. Phase 3.2 can record blocked, idempotent job state but cannot invoke an adapter or launch a process.

Phase 3.3 makes that blocked state durable:

• the Orchestrator owns a PostgreSQL executor_jobs table beside its LangGraph checkpoint tables, under the same isolated schema and database credentials;
• the build_plan node atomically creates one deterministic Executor Job per run and graph-node version;
• retries recover the same job through its idempotency key, while a changed semantic request fails closed;
• /health and /ready report and probe checkpoint and Executor Job storage independently;
• memory mode keeps both stores explicitly non-durable.

The persisted job is still a preview terminal state: status=blocked, reason=executor_dispatch_not_implemented, and executionEnabled=false. Phase 3.3 does not call Goosed, Aider, or OpenHands.

Phase 3.4 adds read-only Executor Job observability:

• executor-job-event-v1 is stored in the separate PostgreSQL executor_job_events table with a per-job sequence cursor;
• initial state and its first event are committed atomically, as are later state transitions and their events;
• existing Phase 3.3 snapshots receive a deterministic executor_job_snapshot_imported migration event;
• the versioned API exposes job state and cursor-paged events without exposing task instructions, credentials, workspace paths, or executor SDK types;
• unsafe or oversized run identifiers are represented by a deterministic SHA-256-derived opaque Executor Job id so PostgreSQL and URL identifiers stay bounded;
• Shadow Run detail in memindadm joins Native state, LangGraph checkpoints, Executor Job state, and Executor Job events;
• Canary readiness now requires both checkpoint and Executor Job storage to be durable.

These endpoints are observational only. They do not add submit, retry, claim, or adapter-dispatch operations.

The default memindadm mode remains off. Canary and Active routing are not wired to the LangGraph executor in Phase 3. Native Agent Run remains the sole executor in every mode.

Admin configuration

The memindadm page is /ops/admin/orchestrator.

Supported modes:

• off: Native Agent Run only.
• shadow: Native executes; the configured engine may observe.
• canary: user allowlist and deterministic rollout percentage produce Dry-run candidate decisions.
• active: workflow-allowlisted tasks produce Dry-run primary-engine decisions.

MEMIND_ORCHESTRATOR_KILL_SWITCH=1 always forces Native selection.

Local process

PostgreSQL is the default and required checkpoint and Executor Job backend:

MEMIND_ORCHESTRATOR_DATABASE_URL='postgresql://...' \
MEMIND_ORCHESTRATOR_SERVICE_TOKEN='...' \
pnpm dev:orchestrator

Memory mode is intentionally opt-in and non-durable:

MEMIND_ORCHESTRATOR_CHECKPOINT_MODE=memory pnpm dev:orchestrator

The service binds to 127.0.0.1:8093 by default. Configure the same URL and service token in Portal, then enable shadow in /ops/admin/orchestrator.

memindadm exposes the projection through:

GET /admin-api/orchestrator/shadow-runs
GET /admin-api/orchestrator/shadow-runs/:runId
GET /admin-api/orchestrator/canary-readiness
GET /admin-api/orchestrator/execution-plans

Selecting a Shadow Run in memindadm also reads the linked Executor Job through the Orchestrator API:

GET /v1/executor-jobs/:jobId
GET /v1/executor-jobs/:jobId/events?after=<cursor>&limit=<limit>

Colima deployment

Colima is the recommended first container host on macOS because this service and its PostgreSQL checkpoint database have an isolated Compose lifecycle:

cd deploy/orchestrator
cp .env.example .env
# Replace both example secrets before starting.
docker compose up -d --build
docker compose ps
curl http://127.0.0.1:8093/ready

When Portal or the Shadow smoke process runs outside Compose, export the same service token into the child process environment. Plain source does not export new shell variables by itself:

set -a
source deploy/orchestrator/.env
set +a
export MEMIND_ORCHESTRATOR_URL=http://127.0.0.1:8093
pnpm dev

The Compose project:

• publishes only the Orchestrator port on loopback;
• does not publish PostgreSQL;
• does not mount the Docker socket or a Memind workspace;
• keeps PostgreSQL on a dedicated internal network while the service also joins a separate edge network for its loopback-published port;
• is not part of the Goosed Compose lifecycle.

This local Compose artifact is not a production release path. Production still requires the repository release gates and a separately approved deployment.

Runtime environment

Variable	Purpose	Default
MEMIND_ORCHESTRATOR_HOST	HTTP bind address	127.0.0.1
MEMIND_ORCHESTRATOR_PORT	HTTP port	8093
MEMIND_ORCHESTRATOR_SERVICE_TOKEN	Bearer token for /v1/*	empty
MEMIND_ORCHESTRATOR_CHECKPOINT_MODE	postgres or explicit memory	postgres
MEMIND_ORCHESTRATOR_DATABASE_URL	Dedicated Orchestrator PostgreSQL URL	required
MEMIND_ORCHESTRATOR_DATABASE_SCHEMA	Checkpoint and Executor Job schema	memind_orchestrator

Extraction test

The service is ready to move into a separate repository only when:

1. it can run against a mock Memind control plane;
2. it owns its checkpoint database and credentials;
3. task inputs use resource references rather than host paths;
4. executor calls use versioned adapters;
5. moving the service requires only changing ORCHESTRATOR_URL.