docs: ml-based prompt injection detection (#6627)
This commit is contained in:
dianed-square
@@ -47,9 +47,10 @@ The following settings can be configured at the root level of your config.yaml f
|
||||
| `OTEL_EXPORTER_OTLP_ENDPOINT` | OTLP endpoint URL for [observability](/docs/guides/environment-variables#opentelemetry-protocol-otlp) | URL (e.g., `http://localhost:4318`) | None | No |
|
||||
| `OTEL_EXPORTER_OTLP_TIMEOUT` | Export timeout in milliseconds for [observability](/docs/guides/environment-variables#opentelemetry-protocol-otlp) | Integer (ms) | 10000 | No |
|
||||
| `SECURITY_PROMPT_ENABLED` | Enable [prompt injection detection](/docs/guides/security/prompt-injection-detection) to identify potentially harmful commands | true/false | false | No |
|
||||
| `SECURITY_PROMPT_THRESHOLD` | Sensitivity threshold for [prompt injection detection](/docs/guides/security/prompt-injection-detection) (higher = stricter) | Float between 0.01 and 1.0 | 0.7 | No |
|
||||
<!-- | `SECURITY_PROMPT_CLASSIFIER_ENABLED` | Enable ML-based prompt injection detection for advanced threat identification | true/false | false | No | -->
|
||||
<!-- | `SECURITY_PROMPT_CLASSIFIER_MODEL` | Specify the BERT ML model to use for prompt injection detection | String | "ProtectAI DeBERTa" | No | -->
|
||||
| `SECURITY_PROMPT_THRESHOLD` | Sensitivity threshold for prompt injection detection (higher = stricter) | Float between 0.01 and 1.0 | 0.8 | No |
|
||||
| `SECURITY_PROMPT_CLASSIFIER_ENABLED` | Enable ML-based prompt injection detection for advanced threat identification | true/false | false | No |
|
||||
| `SECURITY_PROMPT_CLASSIFIER_ENDPOINT` | Classification endpoint URL for ML-based prompt injection detection | URL (e.g., "https://api.example.com/classify") | None | No |
|
||||
| `SECURITY_PROMPT_CLASSIFIER_TOKEN` | Authentication token for `SECURITY_PROMPT_CLASSIFIER_ENDPOINT` | String | None | No |
|
||||
|
||||
## Experimental Features
|
||||
|
||||
|
||||
@@ -295,6 +295,28 @@ These variables control security related features.
|
||||
|----------|---------|---------|---------|
|
||||
| `GOOSE_ALLOWLIST` | Controls which extensions can be loaded | URL for [allowed extensions](/docs/guides/allowlist) list | Unset |
|
||||
| `GOOSE_DISABLE_KEYRING` | Disables the system keyring for secret storage | Set to any value (e.g., "1", "true", "yes") to disable. The actual value doesn't matter, only whether the variable is set. | Unset (keyring enabled) |
|
||||
| `SECURITY_PROMPT_ENABLED` | Enable [prompt injection detection](/docs/guides/security/prompt-injection-detection) to identify potentially harmful commands | true/false | false |
|
||||
| `SECURITY_PROMPT_THRESHOLD` | Sensitivity threshold for prompt injection detection (higher = stricter) | Float between 0.01 and 1.0 | 0.8 |
|
||||
| `SECURITY_PROMPT_CLASSIFIER_ENABLED` | Enable ML-based prompt injection detection for advanced threat identification | true/false | false |
|
||||
| `SECURITY_PROMPT_CLASSIFIER_ENDPOINT` | Classification endpoint URL for ML-based prompt injection detection | URL (e.g., "https://api.example.com/classify") | Unset |
|
||||
| `SECURITY_PROMPT_CLASSIFIER_TOKEN` | Authentication token for `SECURITY_PROMPT_CLASSIFIER_ENDPOINT` | String | Unset |
|
||||
|
||||
**Examples**
|
||||
|
||||
```bash
|
||||
# Enable prompt injection detection with default threshold
|
||||
export SECURITY_PROMPT_ENABLED=true
|
||||
|
||||
# Enable with custom threshold (stricter)
|
||||
export SECURITY_PROMPT_ENABLED=true
|
||||
export SECURITY_PROMPT_THRESHOLD=0.9
|
||||
|
||||
# Enable ML-based detection with external endpoint
|
||||
export SECURITY_PROMPT_ENABLED=true
|
||||
export SECURITY_PROMPT_CLASSIFIER_ENABLED=true
|
||||
export SECURITY_PROMPT_CLASSIFIER_ENDPOINT="https://your-endpoint.com/classify"
|
||||
export SECURITY_PROMPT_CLASSIFIER_TOKEN="your-auth-token"
|
||||
```
|
||||
|
||||
:::tip
|
||||
When the keyring is disabled, secrets are stored here:
|
||||
|
||||
@@ -1,19 +1,24 @@
|
||||
---
|
||||
sidebar_position: 2
|
||||
title: Classification API Specification
|
||||
unlisted: true
|
||||
description: API specification for self-hosting ML-based prompt injection detection endpoints.
|
||||
---
|
||||
|
||||
This document defines the API that Goose uses for ML-based prompt injection detection.
|
||||
This API specification defines the API that goose uses for ML-based [prompt injection detection](/docs/guides/security/prompt-injection-detection).
|
||||
|
||||
## Overview
|
||||
:::info For Self-Hosting Only
|
||||
This API specification is intended as a reference for users who want to self-host their own model and classification endpoint.
|
||||
|
||||
Goose requires a classification endpoint that can analyze text and return a score indicating the likelihood of prompt injection. This API follows the **HuggingFace Inference API format** for text classification, making it compatible with [HuggingFace Inference Endpoints](https://huggingface.co/docs/inference-providers/providers/hf-inference).
|
||||
If you're using an existing inference service like Hugging Face, you can just configure it in your [prompt injection detection](/docs/guides/security/prompt-injection-detection) settings.
|
||||
:::
|
||||
|
||||
goose requires a classification endpoint that can analyze text and return a score indicating the likelihood of prompt injection. This API follows the Hugging Face Inference API format for text classification, making it compatible with [Hugging Face Inference Endpoints](https://huggingface.co/docs/inference-providers/providers/hf-inference).
|
||||
|
||||
## Security & Privacy Considerations
|
||||
**Warning:** When using ML-based prompt injection detection, all tool call content and user messages sent for classification will be transmitted to the configured endpoint. This may include sensitive or confidential information.
|
||||
- If you use an external or third-party endpoint (e.g., HuggingFace Inference API, cloud-hosted models), your data will be sent over the network and processed by that service.
|
||||
- If you use an external or third-party endpoint (e.g., Hugging Face Inference API, cloud-hosted models), your data will be sent over the network and processed by that service.
|
||||
- Consider the sensitivity of your data before enabling ML-based detection or selecting an endpoint.
|
||||
- For highly sensitive or regulated data, use a self-hosted endpoint, run BERT models locally (see reference implementation) or ensure your chosen provider meets your security and compliance requirements.
|
||||
- For highly sensitive or regulated data, use a self-hosted endpoint, run BERT models locally or ensure your chosen provider meets your security and compliance requirements.
|
||||
- Review the endpoint's privacy policy and data handling practices.
|
||||
|
||||
## Endpoint
|
||||
@@ -22,7 +27,7 @@ Goose requires a classification endpoint that can analyze text and return a scor
|
||||
|
||||
Analyzes text for prompt injection and returns classification results.
|
||||
|
||||
**Note:** The endpoint path can be configured. For HuggingFace, it's typically `/models/{model-id}`. For custom implementations, it can be any path (e.g., `/classify`, `/v1/classify`).
|
||||
**Note:** The endpoint path can be configured. For Hugging Face, it's typically `/models/{model-id}`. For custom implementations, it can be any path (e.g., `/classify`, `/v1/classify`).
|
||||
|
||||
#### Request
|
||||
|
||||
@@ -68,17 +73,17 @@ Analyzes text for prompt injection and returns classification results.
|
||||
- `"SAFE"` or `"LABEL_0"`: Indicates safe/benign text
|
||||
- Implementations SHOULD return results sorted by score (highest first)
|
||||
|
||||
**Goose's Usage:**
|
||||
- Goose looks for the label with the highest score
|
||||
- If the top label is "INJECTION" (or "LABEL_1"), the score is used as the injection confidence
|
||||
- If the top label is "SAFE" (or "LABEL_0"), Goose uses `1.0 - score` as the injection confidence
|
||||
**goose's Usage:**
|
||||
- goose looks for the label with the highest score
|
||||
- If the top label is `"INJECTION"` (or `"LABEL_1"`), the score is used as the injection confidence
|
||||
- If the top label is `"SAFE"` (or `"LABEL_0"`), goose uses `1.0 - score` as the injection confidence
|
||||
|
||||
#### Status Codes
|
||||
|
||||
- `200 OK`: Successful classification
|
||||
- `400 Bad Request`: Invalid request format
|
||||
- `500 Internal Server Error`: Classification failed
|
||||
- `503 Service Unavailable`: Model is loading (HuggingFace specific)
|
||||
- `503 Service Unavailable`: Model is loading (Hugging Face specific)
|
||||
|
||||
#### Example
|
||||
|
||||
|
||||
@@ -17,9 +17,14 @@ import styles from '@site/src/components/Card/styles.module.css';
|
||||
<div className={styles.cardGrid}>
|
||||
<Card
|
||||
title="Prompt Injection Detection"
|
||||
description="Detect and prevent potentially harmful commands before they run with pattern-based security scanning."
|
||||
description="Detect and prevent potentially harmful commands before they run."
|
||||
link="/docs/guides/security/prompt-injection-detection"
|
||||
/>
|
||||
<Card
|
||||
title="Classification API Specification"
|
||||
description="API specification for self-hosting ML-based prompt injection detection endpoints."
|
||||
link="/docs/guides/security/classification-api-spec"
|
||||
/>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
|
||||
@@ -1,4 +1,5 @@
|
||||
---
|
||||
sidebar_position: 1
|
||||
title: Prompt Injection Detection
|
||||
sidebar_label: Prompt Injection Detection
|
||||
description: Protect your workflow by detecting potentially harmful commands before they run.
|
||||
@@ -16,15 +17,17 @@ You can help protect your goose workflows by enabling prompt injection detection
|
||||
- Attempts to access or exfiltrate sensitive data like SSH keys
|
||||
- System modifications that could compromise security
|
||||
|
||||
In addition, you can optionally enable [ML-based scanning](#enhanced-detection-with-machine-learning) using a specified model.
|
||||
|
||||
:::important
|
||||
These checks provide a safeguard, not a guarantee. They detect known patterns but cannot catch all possible threats, especially novel or sophisticated attacks.
|
||||
:::
|
||||
|
||||
## How Detection Works
|
||||
|
||||
When enabled, goose scans tool calls for risky patterns before they run:
|
||||
When enabled, goose uses a multi-layered approach to detect threats before they run:
|
||||
|
||||
1. **Tool call is intercepted and analyzed** - When goose prepares to execute a tool, the security system extracts the tool parameter text and checks it against [threat patterns](https://github.com/block/goose/blob/main/crates/goose/src/security/patterns.rs)
|
||||
1. **Tool call is intercepted and analyzed** - When goose prepares to execute a tool, the security system extracts the tool parameter text and checks it against [threat patterns](https://github.com/block/goose/blob/main/crates/goose/src/security/patterns.rs). If ML-based detection is enabled, it also uses machine learning to analyze the semantic content of the tool call and recent conversation messages to better understand context and reduce false positives.
|
||||
2. **Risk is assessed** - Detected threats are assigned confidence scores
|
||||
3. **Execution pauses** - Threats that exceed your configured threshold need your decision
|
||||
4. **Security alert appears** - The alert displays the confidence level, a description of the finding, and a unique finding ID. For example:
|
||||
@@ -60,15 +63,25 @@ When in doubt, deny.
|
||||
3. Click the `Chat` tab
|
||||
4. Toggle `Enable Prompt Injection Detection` to the on setting
|
||||
5. Optionally adjust the `Detection Threshold` to [configure the sensitivity](#configuring-detection-threshold)
|
||||
6. Optionally enable ML-based detection:
|
||||
1. Toggle `Enable ML-based Detection` to the on setting
|
||||
2. Configure your inference endpoint:
|
||||
- `Endpoint URL`: URL to the classification service (e.g., Hugging Face)
|
||||
- `API Token`: Authentication token if required by your service
|
||||
|
||||
</TabItem>
|
||||
<TabItem value="config" label="goose config file">
|
||||
|
||||
Add these settings to your [`config.yaml`](/docs/guides/config-files):
|
||||
Add security prompt settings to your [`config.yaml`](/docs/guides/config-files):
|
||||
|
||||
```yaml
|
||||
SECURITY_PROMPT_ENABLED: true
|
||||
SECURITY_PROMPT_THRESHOLD: 0.7 # Optional, default is 0.7
|
||||
SECURITY_PROMPT_THRESHOLD: 0.8 # Optional, default is 0.8
|
||||
|
||||
# Optional: Enable ML-based detection (Hugging Face example)
|
||||
SECURITY_PROMPT_CLASSIFIER_ENABLED: true
|
||||
SECURITY_PROMPT_CLASSIFIER_ENDPOINT: "https://router.huggingface.co/hf-inference/models/protectai/deberta-v3-base-prompt-injection-v2"
|
||||
SECURITY_PROMPT_CLASSIFIER_TOKEN: "YOUR_HUGGING_FACE_TOKEN"
|
||||
```
|
||||
|
||||
</TabItem>
|
||||
@@ -92,10 +105,27 @@ The threshold (0.01-1.0) controls how strict detection is:
|
||||
| **0.70-0.90** | Strict | Working with sensitive data or systems |
|
||||
| **0.90-1.00** | Maximum | High-security environments |
|
||||
|
||||
When the injection prompt detection feature is enabled, the default threshold is 0.7 (recommended for most users).
|
||||
When the injection prompt detection feature is enabled, the default threshold is 0.8 (recommended for most users).
|
||||
|
||||
Lower thresholds mean fewer alerts but might miss threats. Higher thresholds catch more potential issues but may flag legitimate operations. You can control this sensitivity/convenience tradeoff based on your needs.
|
||||
|
||||
## Enhanced Detection with Machine Learning
|
||||
|
||||
By default, prompt injection detection uses pattern matching, but you can optionally enable ML-based detection for improved accuracy and fewer false positives.
|
||||
|
||||
ML-based detection:
|
||||
- Analyzes the semantic content of tool calls and recent messages
|
||||
- Detects sophisticated attacks that patterns might miss
|
||||
- Reduces false positives by understanding conversation context
|
||||
- Requires providing a classification endpoint URL and API token (if required)
|
||||
|
||||
:::warning Privacy Consideration
|
||||
When ML-based detection is enabled, tool call content and recent messages are sent to the configured endpoint for analysis.
|
||||
:::
|
||||
|
||||
#### Self-Hosting ML Detection Endpoints
|
||||
If you want to run your own classification endpoint, see the [Classification API Specification](/docs/guides/security/classification-api-spec) for implementation details. The API follows the Hugging Face Inference API format.
|
||||
|
||||
## See Also
|
||||
|
||||
- [goose Permission Modes](/docs/guides/goose-permissions) - Control goose's autonomy level
|
||||
|
||||
Reference in New Issue
Block a user