Provider error proxy for simulating various types of errors (#5091)
This commit is contained in:
David Katz
@@ -0,0 +1,220 @@
|
||||
# Provider Error Proxy
|
||||
|
||||
A network-level HTTP proxy for simulating provider errors when testing Goose's error handling and retry logic.
|
||||
|
||||
## Features
|
||||
|
||||
- **Interactive error injection**: Manually trigger different error types via stdin commands
|
||||
- **Network-level interception**: No changes to Goose's Rust code required
|
||||
- **Multi-provider support**: Works with OpenAI, Anthropic, Google, OpenRouter, Tetrate, and Databricks
|
||||
- **Streaming support**: Handles both regular HTTP responses and streaming responses (SSE/chunked)
|
||||
- **Provider-specific errors**: Returns appropriate error codes and formats for each provider
|
||||
- **Transparent proxying**: Forwards all other requests unchanged to the real provider APIs
|
||||
|
||||
## Quickstart
|
||||
|
||||
```bash
|
||||
# 1. Start the proxy (from scripts/provider-error-proxy directory)
|
||||
uv run proxy.py
|
||||
|
||||
# 2. In another terminal, configure Goose to use the proxy
|
||||
export OPENAI_HOST=http://localhost:8888
|
||||
export ANTHROPIC_HOST=http://localhost:8888
|
||||
export GOOGLE_HOST=http://localhost:8888
|
||||
export OPENROUTER_HOST=http://localhost:8888
|
||||
export TETRATE_HOST=http://localhost:8888
|
||||
export DATABRICKS_HOST=http://localhost:8888
|
||||
|
||||
# For Databricks with OAuth, also set the real host:
|
||||
export DATABRICKS_REAL_HOST=https://your-workspace.databricks.com
|
||||
|
||||
# 3. Run Goose normally
|
||||
goose session start "tell me a joke"
|
||||
|
||||
# 4. In the proxy terminal, use interactive commands:
|
||||
# n - No error (pass through) - permanent
|
||||
# c - Context length exceeded error
|
||||
# r - Rate limit error
|
||||
# u - Unknown server error (500)
|
||||
# q - Quit
|
||||
```
|
||||
|
||||
## Installation
|
||||
|
||||
This project uses `uv` for Python dependency management. From the `scripts/provider-error-proxy` directory:
|
||||
|
||||
```bash
|
||||
# Install dependencies (uv will handle this automatically)
|
||||
uv sync
|
||||
```
|
||||
|
||||
## Usage
|
||||
|
||||
### Starting the Proxy
|
||||
|
||||
Start the proxy with default settings (port 8888):
|
||||
|
||||
```bash
|
||||
uv run proxy.py
|
||||
```
|
||||
|
||||
Use a custom port:
|
||||
|
||||
```bash
|
||||
uv run proxy.py --port 9000
|
||||
```
|
||||
|
||||
### Interactive Commands
|
||||
|
||||
Once the proxy is running, you can control error injection interactively:
|
||||
|
||||
- **`n`** - No error (pass through all requests normally) - **permanent mode**
|
||||
- **`c`** - Context length exceeded error (1 time by default)
|
||||
- `c 4` - Inject error 4 times in a row
|
||||
- `c 0.3` or `c 30%` - Inject error on 30% of requests
|
||||
- `c *` - Inject error on 100% of requests (all requests fail)
|
||||
- **`r`** - Rate limit error (1 time by default, same modifiers as `c`)
|
||||
- **`u`** - Unknown server error (500) (1 time by default, same modifiers as `c`)
|
||||
- **`q`** - Quit the proxy
|
||||
|
||||
**Note:** Whitespace is flexible - `c 100%`, `c100%`, `c *`, and `c*` all work the same way.
|
||||
|
||||
The proxy will display the current mode and request count after each command.
|
||||
|
||||
### Configuring Goose
|
||||
|
||||
Set environment variables to redirect provider traffic through the proxy:
|
||||
|
||||
```bash
|
||||
export OPENAI_HOST=http://localhost:8888
|
||||
export ANTHROPIC_HOST=http://localhost:8888
|
||||
export GOOGLE_HOST=http://localhost:8888
|
||||
export OPENROUTER_HOST=http://localhost:8888
|
||||
export TETRATE_HOST=http://localhost:8888
|
||||
export DATABRICKS_HOST=http://localhost:8888
|
||||
```
|
||||
|
||||
For providers that require authentication or metadata endpoints (like Databricks with OAuth), you also need to set the real host:
|
||||
|
||||
```bash
|
||||
export DATABRICKS_REAL_HOST=https://your-workspace.databricks.com
|
||||
```
|
||||
|
||||
Then run Goose normally. The proxy will intercept API requests and you can manually trigger errors as needed, while authentication and metadata requests are forwarded to the real provider.
|
||||
|
||||
## How It Works
|
||||
|
||||
1. **Request Interception**: The proxy listens on localhost and receives all provider API requests
|
||||
2. **Provider Detection**: Identifies which provider the request is for based on headers and paths
|
||||
3. **Smart Forwarding**: Authentication, OIDC, and metadata endpoints are always forwarded to the real provider without error injection
|
||||
4. **Interactive Error Control**: Use stdin commands to control when and what type of errors to inject
|
||||
5. **Error Injection**: When an error mode is active, API requests return provider-specific error responses
|
||||
6. **Streaming Support**: Detects streaming responses (SSE/chunked) and streams them through transparently
|
||||
7. **Transparent Forwarding**: All other requests are forwarded to the actual provider API unchanged
|
||||
|
||||
### Streaming Details
|
||||
|
||||
The proxy automatically detects and handles streaming responses by:
|
||||
- Checking for `text/event-stream` content type (Server-Sent Events)
|
||||
- Using `StreamResponse` to forward chunks in real-time without buffering
|
||||
|
||||
This means streaming completions from providers like OpenAI, Anthropic, and Databricks work seamlessly through the proxy.
|
||||
|
||||
## Error Types by Provider
|
||||
|
||||
The proxy returns realistic error responses for each provider:
|
||||
|
||||
### Context Length Exceeded (Command: `c`)
|
||||
- **OpenAI**: 400 with `context_length_exceeded` error
|
||||
- **Anthropic**: 400 with "prompt is too long" message
|
||||
- **Google**: 400 with `INVALID_ARGUMENT` status
|
||||
- **OpenRouter**: 400 with context length message
|
||||
- **Tetrate**: 400 with context length error
|
||||
- **Databricks**: 400 with `INVALID_PARAMETER_VALUE` error
|
||||
|
||||
### Rate Limit (Command: `r`)
|
||||
- **OpenAI**: 429 with `rate_limit_exceeded` error
|
||||
- **Anthropic**: 429 with `rate_limit_error` type
|
||||
- **Google**: 429 with `RESOURCE_EXHAUSTED` status
|
||||
- **OpenRouter**: 429 with rate limit message
|
||||
- **Tetrate**: 429 with rate limit error
|
||||
- **Databricks**: 429 with `RATE_LIMIT_EXCEEDED` error
|
||||
|
||||
### Server Error (Command: `u`)
|
||||
- **OpenAI**: 500 with `internal_server_error` error
|
||||
- **Anthropic**: 529 with `overloaded_error` type
|
||||
- **Google**: 503 with `UNAVAILABLE` status
|
||||
- **OpenRouter**: 500 with internal server error
|
||||
- **Tetrate**: 503 with service unavailable error
|
||||
- **Databricks**: 500 with `INTERNAL_ERROR` error
|
||||
|
||||
## Example Session
|
||||
|
||||
```
|
||||
$ uv run proxy.py
|
||||
============================================================
|
||||
🔧 Provider Error Proxy
|
||||
============================================================
|
||||
Port: 8888
|
||||
|
||||
To use with Goose, set these environment variables:
|
||||
export OPENAI_HOST=http://localhost:8888
|
||||
export ANTHROPIC_HOST=http://localhost:8888
|
||||
...
|
||||
============================================================
|
||||
|
||||
============================================================
|
||||
Current mode: ✅ No error (pass through)
|
||||
Requests handled: 0
|
||||
============================================================
|
||||
|
||||
Commands:
|
||||
n - No error (pass through) - permanent
|
||||
c - Context length exceeded (1 time)
|
||||
c 4 - Context length exceeded (4 times)
|
||||
c 0.3 - Context length exceeded (30% of requests)
|
||||
c 30% - Context length exceeded (30% of requests)
|
||||
c * - Context length exceeded (100% of requests)
|
||||
r - Rate limit error (1 time)
|
||||
u - Unknown server error (1 time)
|
||||
q - Quit
|
||||
|
||||
Enter command: r
|
||||
============================================================
|
||||
Current mode: ⏱️ Rate limit exceeded (1 remaining)
|
||||
Requests handled: 0
|
||||
============================================================
|
||||
...
|
||||
2025-10-09 14:30:15 - __main__ - INFO - 📨 Request #1: POST /v1/chat/completions -> openai
|
||||
2025-10-09 14:30:15 - __main__ - WARNING - 💥 Injecting RATE_LIMIT error (status 429) for openai
|
||||
|
||||
Enter command: n
|
||||
============================================================
|
||||
Current mode: ✅ No error (pass through)
|
||||
Requests handled: 1
|
||||
============================================================
|
||||
...
|
||||
2025-10-09 14:30:20 - __main__ - INFO - 📨 Request #2: POST /v1/chat/completions -> openai
|
||||
2025-10-09 14:30:20 - __main__ - INFO - ✅ Proxied response: 200
|
||||
```
|
||||
|
||||
## Development
|
||||
|
||||
The proxy is built with `aiohttp` for async HTTP handling. Key components:
|
||||
|
||||
- `ErrorProxy`: Main proxy class that handles request interception and error injection
|
||||
- `ErrorMode`: Enum defining the available error injection modes
|
||||
- `detect_provider()`: Identifies which provider based on headers/paths
|
||||
- `handle_request()`: Main request handler that either proxies or returns errors
|
||||
- `stdin_reader()`: Thread that reads interactive commands from stdin
|
||||
|
||||
## Testing
|
||||
|
||||
To test the proxy:
|
||||
|
||||
1. Start the proxy: `uv run proxy.py`
|
||||
2. Configure Goose to use the proxy (set environment variables)
|
||||
3. Run Goose in another terminal
|
||||
4. Use interactive commands to trigger different error types
|
||||
5. Observe how Goose handles each error type
|
||||
6. Check proxy logs to see which requests were forwarded vs. errored
|
||||
Reference in New Issue
Block a user