memind/docs/deep-search-service.md

# TKMind Deep Search Service

TKMind Deep Search is a local-first, independently deployable research orchestrator. MindSearch treats it as a `research-http` service, in the same way that SearXNG is registered as a search provider.

The standalone runtime uses the built-in `node:sqlite` module and therefore requires a Node.js release that provides `DatabaseSync` (Node 22.5 or newer; the local verification used Node 26).

## Runtime flow

1. Validate and persist the task.
2. Build a depth-aware research plan.
3. Run bounded, multi-round searches through SearXNG.
4. Canonicalize, deduplicate, and rerank source URLs.
5. Resolve DNS and block private destinations before reading public pages.
6. Extract evidence and remove duplicate claims.
7. Generate a Markdown report with numbered citations.
8. Persist task state, sources, events, report, and research memory in SQLite.

The planner and report writer use an optional model selected from TKMind's Unified Model Center. MindSearch stores only the Provider Key ID and model name. Deep Search calls a secret-protected Portal gateway, and the provider API key never leaves the Portal process. If the selected model is unavailable, deterministic planning and citation-safe extractive reporting keep the service operational.

`createDeepSearchEngine` also accepts an explicit `memorySink` callback. It is disabled by default; an in-process TKMind integration can inject `memoryV2.write` without giving the standalone service an unrestricted outbound memory endpoint. Memory sink failures are fail-open and recorded as task events.

## Local run

```bash
TKMIND_DEEP_SEARCH_SEARXNG_URL=http://127.0.0.1:8080/search \
TKMIND_DEEP_SEARCH_DB=.deep-search/research.sqlite \
TKMIND_DEEP_SEARCH_SECRET=local-shared-secret \
npm run dev:deep-search
```

The server listens on `127.0.0.1:20100` by default.

On production host `103`, SearXNG is exposed at
`http://127.0.0.1:20080/search`. Goosed containers reach host services through
`host.docker.internal`; Portal rewrites loopback MindSearch service endpoints
only when constructing the MCP extension environment.

## API

| Method | Path | Purpose |
| --- | --- | --- |
| `GET` | `/health` | Runtime health and task counts |
| `POST` | `/v1/search` | Synchronous normalized search |
| `POST` | `/v1/research` | Start an asynchronous research task |
| `GET` | `/v1/research` | List recent tasks |
| `GET` | `/v1/research/:id` | Get progress, evidence metadata, and report |
| `GET` | `/v1/research/:id/events` | Stream progress as server-sent events |
| `DELETE` | `/v1/research/:id` | Cancel a queued or running task |

Start request:

```json
{
  "question": "分析 AI Agent 市场",
  "depth": "standard",
  "userId": "optional-user-id"
}
```

Start response:

```json
{
  "task_id": "uuid",
  "status": "queued",
  "service": "tkmind-deep-search"
}
```

## Configuration

| Variable | Default | Meaning |
| --- | --- | --- |
| `TKMIND_DEEP_SEARCH_HOST` | `127.0.0.1` | Bind host |
| `TKMIND_DEEP_SEARCH_PORT` | `20100` | HTTP port |
| `TKMIND_DEEP_SEARCH_DB` | `.deep-search/research.sqlite` | SQLite database |
| `TKMIND_DEEP_SEARCH_SEARXNG_URL` | `http://127.0.0.1:8080/search` | Upstream search endpoint |
| `TKMIND_DEEP_SEARCH_SECRET` | empty | Optional `X-Secret-Key` required by non-health routes |
| `TKMIND_DEEP_SEARCH_LLM_GATEWAY_URL` | `http://127.0.0.1:8081/api/internal/deep-search/llm` | Portal LLM gateway |
| `TKMIND_DEEP_SEARCH_LLM_GATEWAY_SECRET` | `TKMIND_DEEP_SEARCH_SECRET` | Portal gateway bearer secret |
| `GITHUB_TOKEN` | empty | GitHub code-search token, stored only in the standalone service environment |

## Isolation and safety

- The default bind address is loopback only.
- An optional service secret protects all non-health endpoints.
- The same secret can authenticate the local service to the Portal LLM gateway.
- MCP forwards the current TKMind user ID as `X-User-Id`.
- Task status, event, and cancellation routes enforce user ownership when a user ID is present.
- Source reading rejects credentials, loopback, link-local, private, multicast, and private-DNS destinations.
- Redirect targets are resolved and validated again.
- Request bodies, result counts, source content, timeouts, and task depths are bounded.
- GitHub Code Search is proxied by Deep Search so the provider token is never
  copied into goosed extension environments.

## Production runtime on 103

Deep Search is released independently from Portal and `memindadm`:

```bash
npm run build:deep-search-runtime
CONFIRMED_CI_SHA="$(git rev-parse HEAD)" \
  bash scripts/release-deep-search-runtime-prod.sh
```

The production layout is:

```text
/Users/john/Project/deep-search-runtime/
├── current -> releases/<release-id>/
├── releases/<release-id>/
├── shared/.env
├── data/research.sqlite
├── logs/deep-search.log
└── backups/<release-id>/
```

The release workflow requires a clean `main` that exactly matches
`origin/main`, a matching confirmed CI SHA, artifact checksum verification,
SQLite online backup, LaunchAgent rollback, host health/search probes, and a
Goose-container reachability probe. It never releases or restarts Portal or
`memindadm`.

The installer creates a random service secret on first install. Add a newly
issued, least-privilege GitHub token to `shared/.env` only after revoking any
token that has appeared in chat or logs.

## MindSearch registration

The built-in disabled service is:

```json
{
  "id": "deep-search",
  "adapter": "research-http",
  "endpoint": "http://127.0.0.1:20100",
  "healthPath": "/health",
  "enabled": false,
  "llmProviderKeyId": "",
  "llmModel": ""
}
```

In `memindadm` → `MindSearch` → `TKMind Deep Search`, select an optional active Provider configuration and one of its models. The list comes directly from Unified Model Center. The `测试 LLM` action checks that exact Provider/model combination before saving the MindSearch configuration. Selecting “不使用 LLM（确定性降级）” leaves both fields empty.

After local verification, enable the service and MindSearch, then select `deep-search` for the `research` route. New Goose sessions receive:

- `tkmind_research`
- `tkmind_research_status`
- `tkmind_research_cancel`

Enabling or publishing this service to production is a separate release action and is not performed by local development or tests.