diff --git a/documentation/docs/guides/config-files.md b/documentation/docs/guides/config-files.md index 6681fbd2..9db94da2 100644 --- a/documentation/docs/guides/config-files.md +++ b/documentation/docs/guides/config-files.md @@ -47,9 +47,10 @@ The following settings can be configured at the root level of your config.yaml f | `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP endpoint URL for [observability](/docs/guides/environment-variables#opentelemetry-protocol-otlp) | URL (e.g., `http://localhost:4318`) | None | No | | `OTEL_EXPORTER_OTLP_TIMEOUT` | Export timeout in milliseconds for [observability](/docs/guides/environment-variables#opentelemetry-protocol-otlp) | Integer (ms) | 10000 | No | | `SECURITY_PROMPT_ENABLED` | Enable [prompt injection detection](/docs/guides/security/prompt-injection-detection) to identify potentially harmful commands | true/false | false | No | -| `SECURITY_PROMPT_THRESHOLD` | Sensitivity threshold for [prompt injection detection](/docs/guides/security/prompt-injection-detection) (higher = stricter) | Float between 0.01 and 1.0 | 0.7 | No | - - +| `SECURITY_PROMPT_THRESHOLD` | Sensitivity threshold for prompt injection detection (higher = stricter) | Float between 0.01 and 1.0 | 0.8 | No | +| `SECURITY_PROMPT_CLASSIFIER_ENABLED` | Enable ML-based prompt injection detection for advanced threat identification | true/false | false | No | +| `SECURITY_PROMPT_CLASSIFIER_ENDPOINT` | Classification endpoint URL for ML-based prompt injection detection | URL (e.g., "https://api.example.com/classify") | None | No | +| `SECURITY_PROMPT_CLASSIFIER_TOKEN` | Authentication token for `SECURITY_PROMPT_CLASSIFIER_ENDPOINT` | String | None | No | ## Experimental Features diff --git a/documentation/docs/guides/environment-variables.md b/documentation/docs/guides/environment-variables.md index 09e9da4f..53e1a85c 100644 --- a/documentation/docs/guides/environment-variables.md +++ b/documentation/docs/guides/environment-variables.md @@ -295,6 +295,28 @@ These variables control security related features. |----------|---------|---------|---------| | `GOOSE_ALLOWLIST` | Controls which extensions can be loaded | URL for [allowed extensions](/docs/guides/allowlist) list | Unset | | `GOOSE_DISABLE_KEYRING` | Disables the system keyring for secret storage | Set to any value (e.g., "1", "true", "yes") to disable. The actual value doesn't matter, only whether the variable is set. | Unset (keyring enabled) | +| `SECURITY_PROMPT_ENABLED` | Enable [prompt injection detection](/docs/guides/security/prompt-injection-detection) to identify potentially harmful commands | true/false | false | +| `SECURITY_PROMPT_THRESHOLD` | Sensitivity threshold for prompt injection detection (higher = stricter) | Float between 0.01 and 1.0 | 0.8 | +| `SECURITY_PROMPT_CLASSIFIER_ENABLED` | Enable ML-based prompt injection detection for advanced threat identification | true/false | false | +| `SECURITY_PROMPT_CLASSIFIER_ENDPOINT` | Classification endpoint URL for ML-based prompt injection detection | URL (e.g., "https://api.example.com/classify") | Unset | +| `SECURITY_PROMPT_CLASSIFIER_TOKEN` | Authentication token for `SECURITY_PROMPT_CLASSIFIER_ENDPOINT` | String | Unset | + +**Examples** + +```bash +# Enable prompt injection detection with default threshold +export SECURITY_PROMPT_ENABLED=true + +# Enable with custom threshold (stricter) +export SECURITY_PROMPT_ENABLED=true +export SECURITY_PROMPT_THRESHOLD=0.9 + +# Enable ML-based detection with external endpoint +export SECURITY_PROMPT_ENABLED=true +export SECURITY_PROMPT_CLASSIFIER_ENABLED=true +export SECURITY_PROMPT_CLASSIFIER_ENDPOINT="https://your-endpoint.com/classify" +export SECURITY_PROMPT_CLASSIFIER_TOKEN="your-auth-token" +``` :::tip When the keyring is disabled, secrets are stored here: diff --git a/documentation/docs/guides/security/classification-api-spec.md b/documentation/docs/guides/security/classification-api-spec.md index f902cc05..acf2dfa6 100644 --- a/documentation/docs/guides/security/classification-api-spec.md +++ b/documentation/docs/guides/security/classification-api-spec.md @@ -1,19 +1,24 @@ --- +sidebar_position: 2 title: Classification API Specification -unlisted: true +description: API specification for self-hosting ML-based prompt injection detection endpoints. --- -This document defines the API that Goose uses for ML-based prompt injection detection. +This API specification defines the API that goose uses for ML-based [prompt injection detection](/docs/guides/security/prompt-injection-detection). -## Overview +:::info For Self-Hosting Only +This API specification is intended as a reference for users who want to self-host their own model and classification endpoint. -Goose requires a classification endpoint that can analyze text and return a score indicating the likelihood of prompt injection. This API follows the **HuggingFace Inference API format** for text classification, making it compatible with [HuggingFace Inference Endpoints](https://huggingface.co/docs/inference-providers/providers/hf-inference). +If you're using an existing inference service like Hugging Face, you can just configure it in your [prompt injection detection](/docs/guides/security/prompt-injection-detection) settings. +::: + +goose requires a classification endpoint that can analyze text and return a score indicating the likelihood of prompt injection. This API follows the Hugging Face Inference API format for text classification, making it compatible with [Hugging Face Inference Endpoints](https://huggingface.co/docs/inference-providers/providers/hf-inference). ## Security & Privacy Considerations **Warning:** When using ML-based prompt injection detection, all tool call content and user messages sent for classification will be transmitted to the configured endpoint. This may include sensitive or confidential information. -- If you use an external or third-party endpoint (e.g., HuggingFace Inference API, cloud-hosted models), your data will be sent over the network and processed by that service. +- If you use an external or third-party endpoint (e.g., Hugging Face Inference API, cloud-hosted models), your data will be sent over the network and processed by that service. - Consider the sensitivity of your data before enabling ML-based detection or selecting an endpoint. -- For highly sensitive or regulated data, use a self-hosted endpoint, run BERT models locally (see reference implementation) or ensure your chosen provider meets your security and compliance requirements. +- For highly sensitive or regulated data, use a self-hosted endpoint, run BERT models locally or ensure your chosen provider meets your security and compliance requirements. - Review the endpoint's privacy policy and data handling practices. ## Endpoint @@ -22,7 +27,7 @@ Goose requires a classification endpoint that can analyze text and return a scor Analyzes text for prompt injection and returns classification results. -**Note:** The endpoint path can be configured. For HuggingFace, it's typically `/models/{model-id}`. For custom implementations, it can be any path (e.g., `/classify`, `/v1/classify`). +**Note:** The endpoint path can be configured. For Hugging Face, it's typically `/models/{model-id}`. For custom implementations, it can be any path (e.g., `/classify`, `/v1/classify`). #### Request @@ -68,17 +73,17 @@ Analyzes text for prompt injection and returns classification results. - `"SAFE"` or `"LABEL_0"`: Indicates safe/benign text - Implementations SHOULD return results sorted by score (highest first) -**Goose's Usage:** -- Goose looks for the label with the highest score -- If the top label is "INJECTION" (or "LABEL_1"), the score is used as the injection confidence -- If the top label is "SAFE" (or "LABEL_0"), Goose uses `1.0 - score` as the injection confidence +**goose's Usage:** +- goose looks for the label with the highest score +- If the top label is `"INJECTION"` (or `"LABEL_1"`), the score is used as the injection confidence +- If the top label is `"SAFE"` (or `"LABEL_0"`), goose uses `1.0 - score` as the injection confidence #### Status Codes - `200 OK`: Successful classification - `400 Bad Request`: Invalid request format - `500 Internal Server Error`: Classification failed -- `503 Service Unavailable`: Model is loading (HuggingFace specific) +- `503 Service Unavailable`: Model is loading (Hugging Face specific) #### Example diff --git a/documentation/docs/guides/security/index.mdx b/documentation/docs/guides/security/index.mdx index 7b026861..1905516d 100644 --- a/documentation/docs/guides/security/index.mdx +++ b/documentation/docs/guides/security/index.mdx @@ -17,9 +17,14 @@ import styles from '@site/src/components/Card/styles.module.css';
+
diff --git a/documentation/docs/guides/security/prompt-injection-detection.md b/documentation/docs/guides/security/prompt-injection-detection.md index 1c3eb3dd..f1e789b9 100644 --- a/documentation/docs/guides/security/prompt-injection-detection.md +++ b/documentation/docs/guides/security/prompt-injection-detection.md @@ -1,4 +1,5 @@ --- +sidebar_position: 1 title: Prompt Injection Detection sidebar_label: Prompt Injection Detection description: Protect your workflow by detecting potentially harmful commands before they run. @@ -16,15 +17,17 @@ You can help protect your goose workflows by enabling prompt injection detection - Attempts to access or exfiltrate sensitive data like SSH keys - System modifications that could compromise security +In addition, you can optionally enable [ML-based scanning](#enhanced-detection-with-machine-learning) using a specified model. + :::important These checks provide a safeguard, not a guarantee. They detect known patterns but cannot catch all possible threats, especially novel or sophisticated attacks. ::: ## How Detection Works -When enabled, goose scans tool calls for risky patterns before they run: +When enabled, goose uses a multi-layered approach to detect threats before they run: -1. **Tool call is intercepted and analyzed** - When goose prepares to execute a tool, the security system extracts the tool parameter text and checks it against [threat patterns](https://github.com/block/goose/blob/main/crates/goose/src/security/patterns.rs) +1. **Tool call is intercepted and analyzed** - When goose prepares to execute a tool, the security system extracts the tool parameter text and checks it against [threat patterns](https://github.com/block/goose/blob/main/crates/goose/src/security/patterns.rs). If ML-based detection is enabled, it also uses machine learning to analyze the semantic content of the tool call and recent conversation messages to better understand context and reduce false positives. 2. **Risk is assessed** - Detected threats are assigned confidence scores 3. **Execution pauses** - Threats that exceed your configured threshold need your decision 4. **Security alert appears** - The alert displays the confidence level, a description of the finding, and a unique finding ID. For example: @@ -60,15 +63,25 @@ When in doubt, deny. 3. Click the `Chat` tab 4. Toggle `Enable Prompt Injection Detection` to the on setting 5. Optionally adjust the `Detection Threshold` to [configure the sensitivity](#configuring-detection-threshold) + 6. Optionally enable ML-based detection: + 1. Toggle `Enable ML-based Detection` to the on setting + 2. Configure your inference endpoint: + - `Endpoint URL`: URL to the classification service (e.g., Hugging Face) + - `API Token`: Authentication token if required by your service - Add these settings to your [`config.yaml`](/docs/guides/config-files): + Add security prompt settings to your [`config.yaml`](/docs/guides/config-files): ```yaml SECURITY_PROMPT_ENABLED: true - SECURITY_PROMPT_THRESHOLD: 0.7 # Optional, default is 0.7 + SECURITY_PROMPT_THRESHOLD: 0.8 # Optional, default is 0.8 + + # Optional: Enable ML-based detection (Hugging Face example) + SECURITY_PROMPT_CLASSIFIER_ENABLED: true + SECURITY_PROMPT_CLASSIFIER_ENDPOINT: "https://router.huggingface.co/hf-inference/models/protectai/deberta-v3-base-prompt-injection-v2" + SECURITY_PROMPT_CLASSIFIER_TOKEN: "YOUR_HUGGING_FACE_TOKEN" ``` @@ -92,10 +105,27 @@ The threshold (0.01-1.0) controls how strict detection is: | **0.70-0.90** | Strict | Working with sensitive data or systems | | **0.90-1.00** | Maximum | High-security environments | -When the injection prompt detection feature is enabled, the default threshold is 0.7 (recommended for most users). +When the injection prompt detection feature is enabled, the default threshold is 0.8 (recommended for most users). Lower thresholds mean fewer alerts but might miss threats. Higher thresholds catch more potential issues but may flag legitimate operations. You can control this sensitivity/convenience tradeoff based on your needs. +## Enhanced Detection with Machine Learning + +By default, prompt injection detection uses pattern matching, but you can optionally enable ML-based detection for improved accuracy and fewer false positives. + +ML-based detection: +- Analyzes the semantic content of tool calls and recent messages +- Detects sophisticated attacks that patterns might miss +- Reduces false positives by understanding conversation context +- Requires providing a classification endpoint URL and API token (if required) + +:::warning Privacy Consideration +When ML-based detection is enabled, tool call content and recent messages are sent to the configured endpoint for analysis. +::: + +#### Self-Hosting ML Detection Endpoints +If you want to run your own classification endpoint, see the [Classification API Specification](/docs/guides/security/classification-api-spec) for implementation details. The API follows the Hugging Face Inference API format. + ## See Also - [goose Permission Modes](/docs/guides/goose-permissions) - Control goose's autonomy level