feat: implement local deep search engine

2026-07-23 22:27:07 +08:00
parent 4e36b98c9e
commit be1e4c18c0
12 changed files with 1717 additions and 16 deletions
@@ -0,0 +1,107 @@
+# TKMind Deep Search Service
+
+TKMind Deep Search is a local-first, independently deployable research orchestrator. MindSearch treats it as a `research-http` service, in the same way that SearXNG is registered as a search provider.
+
+The standalone runtime uses the built-in `node:sqlite` module and therefore requires a Node.js release that provides `DatabaseSync` (Node 22.5 or newer; the local verification used Node 26).
+
+## Runtime flow
+
+1. Validate and persist the task.
+2. Build a depth-aware research plan.
+3. Run bounded, multi-round searches through SearXNG.
+4. Canonicalize, deduplicate, and rerank source URLs.
+5. Resolve DNS and block private destinations before reading public pages.
+6. Extract evidence and remove duplicate claims.
+7. Generate a Markdown report with numbered citations.
+8. Persist task state, sources, events, report, and research memory in SQLite.
+
+The planner and report writer use an OpenAI-compatible model when configured. If the model is unavailable, deterministic planning and citation-safe extractive reporting keep the service operational.
+
+`createDeepSearchEngine` also accepts an explicit `memorySink` callback. It is disabled by default; an in-process TKMind integration can inject `memoryV2.write` without giving the standalone service an unrestricted outbound memory endpoint. Memory sink failures are fail-open and recorded as task events.
+
+## Local run
+
+```bash
+TKMIND_DEEP_SEARCH_SEARXNG_URL=http://127.0.0.1:8080/search \
+TKMIND_DEEP_SEARCH_DB=.deep-search/research.sqlite \
+npm run dev:deep-search
+```
+
+The server listens on `127.0.0.1:20100` by default.
+
+## API
+
+| Method | Path | Purpose |
+| --- | --- | --- |
+| `GET` | `/health` | Runtime health and task counts |
+| `POST` | `/v1/search` | Synchronous normalized search |
+| `POST` | `/v1/research` | Start an asynchronous research task |
+| `GET` | `/v1/research` | List recent tasks |
+| `GET` | `/v1/research/:id` | Get progress, evidence metadata, and report |
+| `GET` | `/v1/research/:id/events` | Stream progress as server-sent events |
+| `DELETE` | `/v1/research/:id` | Cancel a queued or running task |
+
+Start request:
+
+```json
+{
+  "question": "分析 AI Agent 市场",
+  "depth": "standard",
+  "userId": "optional-user-id"
+}
+```
+
+Start response:
+
+```json
+{
+  "task_id": "uuid",
+  "status": "queued",
+  "service": "tkmind-deep-search"
+}
+```
+
+## Configuration
+
+| Variable | Default | Meaning |
+| --- | --- | --- |
+| `TKMIND_DEEP_SEARCH_HOST` | `127.0.0.1` | Bind host |
+| `TKMIND_DEEP_SEARCH_PORT` | `20100` | HTTP port |
+| `TKMIND_DEEP_SEARCH_DB` | `.deep-search/research.sqlite` | SQLite database |
+| `TKMIND_DEEP_SEARCH_SEARXNG_URL` | `http://127.0.0.1:8080/search` | Upstream search endpoint |
+| `TKMIND_DEEP_SEARCH_SECRET` | empty | Optional `X-Secret-Key` required by non-health routes |
+| `TKMIND_DEEP_SEARCH_LLM_URL` | empty | OpenAI-compatible chat-completions endpoint |
+| `TKMIND_DEEP_SEARCH_LLM_API_KEY` | empty | Optional model API key |
+| `TKMIND_DEEP_SEARCH_LLM_MODEL` | empty | Planner and report model |
+
+## Isolation and safety
+
+- The default bind address is loopback only.
+- An optional service secret protects all non-health endpoints.
+- MCP forwards the current TKMind user ID as `X-User-Id`.
+- Task status, event, and cancellation routes enforce user ownership when a user ID is present.
+- Source reading rejects credentials, loopback, link-local, private, multicast, and private-DNS destinations.
+- Redirect targets are resolved and validated again.
+- Request bodies, result counts, source content, timeouts, and task depths are bounded.
+
+## MindSearch registration
+
+The built-in disabled service is:
+
+```json
+{
+  "id": "deep-search",
+  "adapter": "research-http",
+  "endpoint": "http://127.0.0.1:20100",
+  "healthPath": "/health",
+  "enabled": false
+}
+```
+
+After local verification, enable the service and MindSearch in `memindadm`, then select `deep-search` for the `research` route. New Goose sessions receive:
+
+- `tkmind_research`
+- `tkmind_research_status`
+- `tkmind_research_cancel`
+
+Enabling or publishing this service to production is a separate release action and is not performed by local development or tests.