merge: integrate langgraph execution runtime
# Conflicts: # agent-run-gateway.test.mjs # capabilities.mjs # package.json # server.mjs
This commit is contained in:
john
@@ -0,0 +1,237 @@
|
||||
# Memind Workflow Orchestrator Boundary
|
||||
|
||||
## Decision
|
||||
|
||||
The Orchestrator starts in the Memind repository but is designed as an
|
||||
independently deployable bounded context. Source colocation does not authorize
|
||||
in-process execution inside Portal.
|
||||
|
||||
The stable boundary is:
|
||||
|
||||
```text
|
||||
Memind Control Plane
|
||||
-> versioned Orchestrator API / events
|
||||
Workflow Orchestrator
|
||||
-> versioned Executor Gateway
|
||||
Executor adapters
|
||||
-> Goosed / Aider / OpenHands
|
||||
```
|
||||
|
||||
LangGraph remains an internal workflow engine implementation. Memind must not
|
||||
depend on LangGraph graph, checkpoint, command, or interrupt types.
|
||||
|
||||
## Ownership
|
||||
|
||||
| Data | Owner |
|
||||
|---|---|
|
||||
| User, authorization, billing, product run | Memind |
|
||||
| Workflow node state, checkpoint, interrupt | Orchestrator |
|
||||
| Goosed session | Goosed plus Session Broker mapping |
|
||||
| Executor job | Executor Gateway / executor adapter |
|
||||
| Workspace and deliverable | Workspace / Artifact / MindSpace services |
|
||||
| User-visible progress and audit | Memind run-event projection |
|
||||
|
||||
The Orchestrator must not read or write Memind user, billing, capability, or
|
||||
provider-key tables. A shared PostgreSQL instance is acceptable during the
|
||||
single-host phase, but the Orchestrator must use its own database and database
|
||||
credentials.
|
||||
|
||||
## Runtime modes
|
||||
|
||||
memindadm owns the versioned routing configuration:
|
||||
|
||||
| Mode | Behavior |
|
||||
|---|---|
|
||||
| `off` | Native Agent Run only |
|
||||
| `shadow` | Native executes; LangGraph observes and compares decisions |
|
||||
| `canary` | Explicit users and deterministic percentage may use LangGraph |
|
||||
| `active` | Workflow-allowlisted tasks use the primary engine |
|
||||
|
||||
All modes support an explicit Native fallback. The environment kill switch
|
||||
`MEMIND_ORCHESTRATOR_KILL_SWITCH=1` overrides admin configuration and forces
|
||||
Native selection.
|
||||
|
||||
The initial workflow allowlist contains only `code-run-v1`. Ordinary chat and
|
||||
existing Goosed session traffic remain outside the Orchestrator path.
|
||||
|
||||
Phase 2 implemented only the `off` and `shadow` execution semantics. Canary and
|
||||
Active were initially planning-only so an administrative configuration mistake
|
||||
could not create two task executors.
|
||||
|
||||
Phase 3 makes that restriction explicit in the contract. Routing returns both
|
||||
`candidateEngine` and `engine`: Canary or Active may nominate LangGraph, while
|
||||
`engine` remains Native. The decision is projected as
|
||||
`workflow_execution_planned` with `dryRun=true` and `handoffAllowed=false`.
|
||||
The execution adapter normalizes authorization, idempotency, timeout,
|
||||
cancellation, and fallback controls. Phase 5 can transfer ownership only when
|
||||
the memindadm switch, Portal environment gate, Orchestrator environment gate,
|
||||
adapter enablement, worker health, authorization, and admission policy all
|
||||
agree. Any missing gate keeps Native effective.
|
||||
|
||||
The Dry-run projection records every code-run decision while the configured
|
||||
mode is Shadow, Canary, or Active, including `canary_not_selected`. In Shadow,
|
||||
the same allowlist and deterministic percentage are evaluated in parallel with
|
||||
the remote observation, so collecting routing evidence does not interrupt
|
||||
Shadow readiness samples. This prevents a selected-only dataset from reporting
|
||||
a meaningless 100% candidate rate. The Memind control plane joins each decision
|
||||
to the Native run terminal state and aggregates candidate rate, selection
|
||||
reasons, task types, session coverage, and Native terminal coverage without
|
||||
querying LangGraph.
|
||||
|
||||
## Protocol
|
||||
|
||||
The framework-neutral contracts are:
|
||||
|
||||
- `orchestrator-run-v1`
|
||||
- `orchestrator-event-v1`
|
||||
- `workflow-execution-request-v1`
|
||||
- `workflow-execution-decision-v1`
|
||||
- `executor-job-request-v1`
|
||||
- `executor-job-state-v1`
|
||||
- `executor-dispatch-decision-v1`
|
||||
|
||||
The Phase 3.2 Executor Gateway owns the adapter registry and executor job state
|
||||
transition boundary. Requests contain workspace/artifact references,
|
||||
authorization and side-effect policy, timeout, cancellation, fallback, and an
|
||||
idempotency key. The store interface requires atomic create-if-absent semantics
|
||||
so retries return the original job and conflicting payloads fail closed.
|
||||
|
||||
Goosed, Aider, and OpenHands began as disabled `contract-only` adapters. The
|
||||
reference in-memory store remains non-durable and is not a deployment backend.
|
||||
In the current Phase 5 implementation the dispatch capability exists, but
|
||||
execution and every adapter remain disabled by default. Shadow jobs still record
|
||||
terminal `blocked` state without launching a process or touching a workspace.
|
||||
|
||||
Phase 3.3 persists that state in the Orchestrator-owned PostgreSQL database.
|
||||
LangGraph checkpoint tables and `executor_jobs` share the isolated
|
||||
`memind_orchestrator` schema and credentials, but remain behind separate
|
||||
storage interfaces and readiness probes. This is a deployment simplification,
|
||||
not a domain ownership leak: neither store can access Memind business tables,
|
||||
and the Executor Job Store can move to a separate database without changing
|
||||
the graph or the Memind-facing protocol.
|
||||
|
||||
The `build_plan` node creates the deterministic preview job
|
||||
`<runId>:executor-preview` with idempotency key
|
||||
`<runId>:build_plan:v1`. The stored result is terminal `blocked` state with
|
||||
`executor_dispatch_not_implemented`; it is a durable audit and recovery
|
||||
boundary, not an execution claim. If the run identifier is unsafe or would
|
||||
overflow the 128-character Job id boundary, the graph substitutes a stable
|
||||
SHA-256-derived opaque prefix.
|
||||
|
||||
Phase 3.4 adds the append-only `executor-job-event-v1` contract. Job snapshots
|
||||
remain in `executor_jobs`, while cursor-addressable events live in
|
||||
`executor_job_events`. A transaction commits the initial snapshot and event
|
||||
together; later state transitions lock the job row and commit the updated
|
||||
snapshot and next event together. This prevents a visible state transition
|
||||
without its corresponding audit event.
|
||||
|
||||
The read boundary is deliberately narrower than an execution control API:
|
||||
|
||||
```text
|
||||
GET /v1/executor-jobs/:jobId
|
||||
GET /v1/executor-jobs/:jobId/events?after=<cursor>&limit=<limit>
|
||||
```
|
||||
|
||||
Phase 3.4 had no HTTP submit, retry, claim, or executor cancel endpoint.
|
||||
Phase 3.5 adds a separately authenticated worker protocol. The control-plane
|
||||
read projection still contains no task instruction, secret, absolute path, lease
|
||||
token, or executor SDK object; only a successful worker claim receives the
|
||||
normalized request and lease token.
|
||||
|
||||
The implemented internal API is:
|
||||
|
||||
```text
|
||||
POST /v1/runs
|
||||
GET /v1/runs/:id
|
||||
POST /v1/runs/:id/resume
|
||||
POST /v1/runs/:id/cancel
|
||||
GET /v1/runs/:id/events?after=<cursor>
|
||||
GET /v1/executor-jobs/:jobId
|
||||
GET /v1/executor-jobs/:jobId/events?after=<cursor>&limit=<limit>
|
||||
```
|
||||
|
||||
External state changes must use an idempotency key derived from run, graph
|
||||
version, node, and attempt. Graph state stores resource references and decisions,
|
||||
not API keys, large logs, binary artifacts, or absolute production paths.
|
||||
|
||||
The graph keeps three deterministic control-plane nodes:
|
||||
|
||||
```text
|
||||
validate_run -> build_plan -> finalize_run
|
||||
```
|
||||
|
||||
Observe-only runs require `sideEffectsAllowed=false`, project the Native
|
||||
boundary, and persist a blocked Executor Job. Explicit active runs require
|
||||
`sideEffectsAllowed=true` plus every execution gate; they queue a job and enter
|
||||
`waiting` until the worker-owned job reaches a terminal state. LangGraph never
|
||||
mounts a workspace or imports an executor SDK.
|
||||
|
||||
## Phase 5 worker and loop boundary
|
||||
|
||||
The task loop is split across durable owners:
|
||||
|
||||
```text
|
||||
LangGraph run: validate -> queue -> wait -> project terminal
|
||||
Executor Job: queued -> leased -> running -> terminal/retryable
|
||||
Worker: claim -> heartbeat -> adapter -> artifact references
|
||||
```
|
||||
|
||||
PostgreSQL row locks and lease tokens fence duplicate claims and late worker
|
||||
updates. Expired leases recover to `retryable` until `maxAttempts` is exhausted,
|
||||
then become `timed_out`. Worker heartbeats are persisted independently of job
|
||||
heartbeats so readiness and autoscaling signals also work while the queue is
|
||||
empty.
|
||||
|
||||
Goosed is a remote HTTP/SSE adapter. Aider and OpenHands are process adapters
|
||||
intended to run in separate worker containers with workspace aliases, an
|
||||
allowlisted writable root, no shell, bounded environment/output, resource
|
||||
limits, and no Docker socket. All adapter outputs cross the boundary as bounded
|
||||
events and artifact references.
|
||||
|
||||
Admission is defense in depth: tenant/user/workspace allowlists, global and
|
||||
per-subject concurrency, and per-minute limits are evaluated before queueing.
|
||||
The public metrics endpoint exposes queue states, claimable jobs, expired
|
||||
leases, worker health, and execution-gate state without task contents.
|
||||
|
||||
## Failure isolation
|
||||
|
||||
Portal invokes Shadow only after the product run and its `queued` event have
|
||||
been committed. The observation is scheduled without awaiting it. A timeout or
|
||||
failure:
|
||||
|
||||
1. cannot reject or delay `createRun`;
|
||||
2. cannot mutate the Native run status;
|
||||
3. is projected as `workflow_shadow_failed`;
|
||||
4. remains removable by deleting the observer wiring and changing only the
|
||||
service URL boundary.
|
||||
|
||||
Successful observations are projected as `workflow_shadow_completed`; graph
|
||||
checkpoints and blocked Executor Jobs stay in the Orchestrator-owned PostgreSQL
|
||||
database.
|
||||
|
||||
Both terminal projection events contain bounded observation latency. The
|
||||
Memind-owned observability service aggregates those events and may join them to
|
||||
the product run status. Only per-run detail calls cross the service boundary to
|
||||
read LangGraph checkpoint state and node events.
|
||||
|
||||
Canary readiness is also a Memind control-plane projection. It excludes
|
||||
synthetic smoke runs and evaluates operational health, durable checkpoint state,
|
||||
durable Executor Job storage, sample volume, session coverage, success rate,
|
||||
latency coverage, P95 latency, Native terminal coverage, and sample freshness.
|
||||
The result is advisory:
|
||||
`manual_canary_review` never mutates routing configuration or transfers
|
||||
execution ownership.
|
||||
|
||||
## Deployment evolution
|
||||
|
||||
1. Local native Orchestrator process with an explicit MemorySaver for debugging.
|
||||
2. Colima Compose with an isolated PostgreSQL checkpoint database.
|
||||
3. Approved server-side shadow service; no production task claim.
|
||||
4. User-allowlisted code-run canary with Native rollback.
|
||||
5. Isolated Aider/OpenHands executor workers.
|
||||
6. Move the same service contract to Linux or Kubernetes when horizontal scaling
|
||||
or independent ownership becomes necessary.
|
||||
|
||||
The Orchestrator must not share the Goosed Compose lifecycle or mount the Docker
|
||||
socket. It calls MindSpace through its service API and calls Goosed through the
|
||||
existing proxy boundary.
|
||||
+41
-27
@@ -1,37 +1,51 @@
|
||||
# Local Memind + Umami analytics
|
||||
# Local Memind analytics (Umami + Rybbit)
|
||||
|
||||
The integration is local-only by default. Memind proxies `/analytics/*` to the
|
||||
local Umami service at `http://127.0.0.1:3100`; it does not contact 103/105.
|
||||
## Umami (optional)
|
||||
|
||||
1. Start `/Users/john/Project/memind-analytics` and verify:
|
||||
Memind can still proxy `/analytics/*` to a local Umami service at
|
||||
`http://127.0.0.1:3100`.
|
||||
|
||||
```bash
|
||||
curl --fail http://127.0.0.1:3100/api/heartbeat
|
||||
```
|
||||
```dotenv
|
||||
MEMIND_ANALYTICS_ENABLED=true
|
||||
MEMIND_ANALYTICS_URL=http://127.0.0.1:3100
|
||||
MEMIND_ANALYTICS_WEBSITE_ID=<website-id>
|
||||
MEMIND_ANALYTICS_ID_SECRET=<random-local-secret>
|
||||
MEMIND_ANALYTICS_DOMAINS=127.0.0.1,localhost
|
||||
```
|
||||
|
||||
2. Create one Umami Website for the local generated-page host. Do not create a
|
||||
Website per page or per user.
|
||||
## Rybbit (recommended for behavior analytics)
|
||||
|
||||
3. Put the Website ID and a local-only pseudonymization secret in Memind's
|
||||
`.env`:
|
||||
Rybbit runs on 105 as `https://rybbit.tkmind.cn`. Local Memind does not talk to
|
||||
103 for analytics. It proxies same-origin `/rybbit/*` to Rybbit `/api/*` so the
|
||||
browser script stays first-party:
|
||||
|
||||
```text
|
||||
Generated Page
|
||||
-> /rybbit/script.js -> https://rybbit.tkmind.cn/api/script.js
|
||||
-> /rybbit/track -> https://rybbit.tkmind.cn/api/track
|
||||
```
|
||||
|
||||
1. In Rybbit, create one Site for the local host you use (`127.0.0.1` or
|
||||
`localhost`). Do not create a Site per page or per user.
|
||||
2. Put the numeric Site ID and a local pseudonymization secret in Memind `.env`:
|
||||
|
||||
```dotenv
|
||||
MEMIND_ANALYTICS_ENABLED=true
|
||||
MEMIND_ANALYTICS_URL=http://127.0.0.1:3100
|
||||
MEMIND_ANALYTICS_WEBSITE_ID=<website-id>
|
||||
MEMIND_ANALYTICS_ID_SECRET=<random-local-secret>
|
||||
MEMIND_ANALYTICS_DOMAINS=127.0.0.1,localhost
|
||||
MEMIND_RYBBIT_ENABLED=true
|
||||
MEMIND_RYBBIT_URL=https://rybbit.tkmind.cn
|
||||
MEMIND_RYBBIT_SITE_ID=2
|
||||
MEMIND_RYBBIT_ID_SECRET=<random-local-secret>
|
||||
```
|
||||
|
||||
4. Restart the local Memind server. Full generated HTML pages will receive a
|
||||
same-origin `/analytics/script.js` tracker. The tracker identifies the
|
||||
visitor with a stable pseudonymous owner ID before sending a standard Umami
|
||||
page view, so Users and Pageviews are populated. The Identify properties
|
||||
include the readable Memind username and current public page URL for
|
||||
operational analytics. Click, form, scroll, and engagement events retain the
|
||||
pseudonymous `owner_id`, `page_id`, and `channel` dimensions.
|
||||
3. Restart the local Memind server. Full generated HTML pages receive a
|
||||
same-origin `/rybbit/script.js` tracker. The tracker identifies the visitor
|
||||
with a stable pseudonymous owner ID, then records engagement events such as
|
||||
`page_click`, scroll depth, and dwell time. Initial pageviews come from
|
||||
Rybbit's site setting `trackInitialPageView`.
|
||||
|
||||
The integration is fail-open: missing configuration, disabled analytics, or a
|
||||
down Umami service leaves page generation and page delivery unchanged. User
|
||||
facing analytics must be queried through a future Memind API that filters by
|
||||
the authenticated owner; do not expose the Umami dashboard directly to users.
|
||||
4. Open Rybbit from local `memind_adm` → Analytics 配置 →「打开 Rybbit 分析后台」
|
||||
(SSO). That path requires matching `MEMIND_RYBBIT_SSO_SECRET` /
|
||||
`RYBBIT_SSO_EMAIL` with the 105 Rybbit deployment.
|
||||
|
||||
Both integrations are fail-open: missing configuration or a down analytics
|
||||
service leaves page generation and page delivery unchanged. Do not expose the
|
||||
Rybbit or Umami dashboards directly to end users.
|
||||
|
||||
@@ -0,0 +1,131 @@
|
||||
# Workflow Orchestrator Phase 5 Runbook
|
||||
|
||||
## Scope
|
||||
|
||||
Phase 5 makes the LangGraph control plane operationally deployable without
|
||||
making it the default executor. The default remains:
|
||||
|
||||
```text
|
||||
memindadm executionEnabled=false
|
||||
Portal MEMIND_ORCHESTRATOR_EXECUTION_HANDOFF_ENABLED=0
|
||||
Orchestrator MEMIND_ORCHESTRATOR_EXECUTION_ENABLED=0
|
||||
enabled executor list empty
|
||||
Native Agent Run owns execution
|
||||
```
|
||||
|
||||
An execution handoff requires all gates to agree: memindadm mode and explicit
|
||||
execution switch, Portal environment gate, deterministic user/workflow rollout,
|
||||
healthy Orchestrator storage, Orchestrator execution gate, enabled adapter,
|
||||
authorization, workspace alias policy, quota admission, and a healthy worker.
|
||||
|
||||
## Local Colima canary
|
||||
|
||||
1. Copy `deploy/orchestrator/.env.example` to the ignored local `.env`.
|
||||
2. Set distinct PostgreSQL, service, and worker secrets.
|
||||
3. Keep execution disabled and start the durable control plane:
|
||||
|
||||
```bash
|
||||
docker compose -f deploy/orchestrator/compose.yaml up -d --build
|
||||
curl -fsS http://127.0.0.1:8093/ready
|
||||
curl -fsS http://127.0.0.1:8093/metrics
|
||||
```
|
||||
|
||||
4. Configure a workspace alias understood by the target Goosed instance.
|
||||
5. Enable only Goosed for a single local user and start the worker profile:
|
||||
|
||||
```text
|
||||
MEMIND_ORCHESTRATOR_EXECUTION_ENABLED=1
|
||||
MEMIND_ORCHESTRATOR_ENABLED_EXECUTORS=goosed
|
||||
MEMIND_ORCHESTRATOR_USER_ALLOWLIST=<local-user-id>
|
||||
MEMIND_EXECUTOR_WORKSPACE_ALIASES_JSON={"canary":"/absolute/canary/path"}
|
||||
```
|
||||
|
||||
```bash
|
||||
docker compose -f deploy/orchestrator/compose.yaml --profile goosed-worker up -d
|
||||
```
|
||||
|
||||
The Orchestrator `/ready` endpoint intentionally fails while execution is
|
||||
enabled and no non-stale worker heartbeat exists.
|
||||
|
||||
## Rollout order
|
||||
|
||||
1. `off`: verify storage, metrics, backup, and worker registration.
|
||||
2. `shadow`: collect Native versus LangGraph planning evidence.
|
||||
3. `canary`, execution switch off: verify selection denominator and readiness.
|
||||
4. `canary`, execution switch on: one explicit user, one workspace alias, zero
|
||||
percentage rollout.
|
||||
5. Increase percentage only after success rate, timeout, expired lease, queue
|
||||
age, fallback, and Native rollback signals remain within the approved SLO.
|
||||
6. `active` remains workflow-allowlisted and requires a separate release
|
||||
approval.
|
||||
|
||||
Never enable Aider or OpenHands in the Orchestrator service until its dedicated
|
||||
worker is registered and healthy. Use the example worker Compose overlay as a
|
||||
template; each tool gets a separate image, resource limits, writable workspace
|
||||
root, read-only container root, no Docker socket, dropped capabilities, and a
|
||||
distinct worker identity.
|
||||
|
||||
## Immediate rollback
|
||||
|
||||
Use any one of these independent controls:
|
||||
|
||||
1. Set `MEMIND_ORCHESTRATOR_KILL_SWITCH=1` on Portal.
|
||||
2. Clear the memindadm execution switch or set mode to `off`.
|
||||
3. Set Portal `MEMIND_ORCHESTRATOR_EXECUTION_HANDOFF_ENABLED=0`.
|
||||
4. Set Orchestrator `MEMIND_ORCHESTRATOR_EXECUTION_ENABLED=0`.
|
||||
5. Remove an executor from `MEMIND_ORCHESTRATOR_ENABLED_EXECUTORS`.
|
||||
6. Drain a worker by stopping it gracefully; the worker records `draining=true`.
|
||||
|
||||
Queued jobs stay durable. Running jobs stop receiving heartbeats, and lease
|
||||
recovery moves them to `retryable` or `timed_out` according to attempt limits.
|
||||
Native fallback remains a Portal decision; the Orchestrator never launches a
|
||||
second Native run by itself.
|
||||
|
||||
## Monitoring
|
||||
|
||||
Scrape `/metrics` and alert on:
|
||||
|
||||
- `memind_orchestrator_executor_expired_leases > 0`;
|
||||
- claimable jobs increasing while healthy workers are zero;
|
||||
- repeated `retryable`, `failed`, or `timed_out` states;
|
||||
- worker heartbeat age beyond 60 seconds;
|
||||
- any execution-enabled interval without durable PostgreSQL readiness.
|
||||
|
||||
Executor events contain bounded metadata and artifact references. They must not
|
||||
contain provider keys, absolute host paths, full stdout/stderr, or binary data.
|
||||
|
||||
## Backup and disaster recovery
|
||||
|
||||
Create and verify a PostgreSQL custom-format backup:
|
||||
|
||||
```bash
|
||||
node scripts/orchestrator-dr.mjs backup --output /safe/path/orchestrator.dump
|
||||
node scripts/orchestrator-dr.mjs verify --input /safe/path/orchestrator.dump
|
||||
```
|
||||
|
||||
Restore only into a confirmed empty target:
|
||||
|
||||
```bash
|
||||
node scripts/orchestrator-dr.mjs restore \
|
||||
--input /safe/path/orchestrator.dump \
|
||||
--database-url postgresql://... \
|
||||
--confirm-empty-target
|
||||
```
|
||||
|
||||
The restore command never uses `--clean` and refuses remote targets unless
|
||||
`--allow-remote-target` is explicitly supplied. After restore, keep execution
|
||||
disabled, start one worker, verify checkpoints/jobs/events/worker heartbeats,
|
||||
then repeat the rollout order from `off`.
|
||||
|
||||
## Production release gate
|
||||
|
||||
This runbook does not authorize a production action. Before any server-side
|
||||
deployment, follow `ENGINEERING_WORKFLOW_RULES.md`,
|
||||
`PRODUCTION_RELEASE_RULES.md`, and run:
|
||||
|
||||
```bash
|
||||
bash scripts/check-release-ready.sh
|
||||
```
|
||||
|
||||
Production still requires a clean commit on complete `main`, passing CI, an
|
||||
explicit deployment approval, a current backup, and a tested rollback.
|
||||
Reference in New Issue
Block a user