tkmind/tkmind_go
1
0
Fork 0
Code Issues Pull Requests Actions 4 Packages Projects Releases Wiki Activity

fix: VMware Tanzu Platform provider - bug fixes, streaming, UI improvements (#8126)

Browse Source 
Signed-off-by: Nick Kuhn <nick.kuhn@broadcom.com>
Signed-off-by: Douwe Osinga <douwe@squareup.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Douwe Osinga <douwe@squareup.com>
This commit is contained in:
Nick Kuhn
2026-03-26 14:16:01 -04:00
committed by GitHub
parent cdf91ea798
commit c936514014
16 changed files with 725 additions and 61 deletions
Download Patch File Download Diff File Expand all files Collapse all files
documentation/docs/getting-started/providers.md
+1 -1
View File
@@ -43,7 +43,7 @@ goose is compatible with a wide range of LLM providers, allowing you to choose a
| [OVHcloud AI](https://www.ovhcloud.com/en/public-cloud/ai-endpoints/) | Provides access to open-source models including Qwen, Llama, Mistral, and DeepSeek through AI Endpoints service. | `OVHCLOUD_API_KEY` |
| [Ramalama](https://ramalama.ai/) | Local model using native [OCI](https://opencontainers.org/) container runtimes, [CNCF](https://www.cncf.io/) tools, and supporting models as OCI artifacts. Ramalama API is a compatible alternative to Ollama and can be used with the goose Ollama provider. Supports Qwen, Llama, DeepSeek, and other open-source models. **Because this provider runs locally, you must first [download and run a model](#local-llms).** | `OLLAMA_HOST` |
| [Snowflake](https://docs.snowflake.com/user-guide/snowflake-cortex/aisql#choosing-a-model) | Access the latest models using Snowflake Cortex services, including Claude models. **Requires a Snowflake account and programmatic access token (PAT)**. | `SNOWFLAKE_HOST`, `SNOWFLAKE_TOKEN` |
| [Tanzu AI Services](https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/ai-services/10-3/ai/index.html) | Enterprise-managed LLM access through VMware Tanzu Platform AI Services. Models are fetched dynamically from the endpoint. | `TANZU_AI_API_KEY`, `TANZU_AI_ENDPOINT` |
| [VMware Tanzu Platform](https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/ai-services/10-3/ai/index.html) | Enterprise-managed LLM access through AI Services on VMware Tanzu Platform. Models are fetched dynamically from the endpoint. | `TANZU_AI_API_KEY`, `TANZU_AI_ENDPOINT` |
| [Tetrate Agent Router Service](https://router.tetrate.ai) | Unified API gateway for AI models including Claude, Gemini, GPT, open-weight models, and others. Supports PKCE authentication flow for secure API key generation. | `TETRATE_API_KEY`, `TETRATE_HOST` (optional) |
| [Venice AI](https://venice.ai/home) | Provides access to open source models like Llama, Mistral, and Qwen while prioritizing user privacy. **Requires an account and an [API key](https://docs.venice.ai/overview/guides/generating-api-key)**. | `VENICE_API_KEY`, `VENICE_HOST` (optional), `VENICE_BASE_PATH` (optional), `VENICE_MODELS_PATH` (optional) |
| [Cerebras](https://cerebras.ai/) | Fast inference on Cerebras wafer-scale engines with models like Llama, Qwen, and others. | `CEREBRAS_API_KEY` |
documentation/docs/guides/tanzu-ai-services.md
+245
View File
@@ -0,0 +1,245 @@
---
sidebar_position: 15
title: VMware Tanzu Platform
description: Connect goose to VMware Tanzu Platform AI Services
---
import Tabs from '@theme/Tabs';
import TabItem from '@theme/TabItem';
# VMware Tanzu Platform
[VMware Tanzu Platform](https://techdocs.broadcom.com/us/en/vmware-tanzu/platform/ai-services/10-3/ai/index.html) provides enterprise-managed LLM access through AI Services. goose connects to VMware Tanzu Platform as an OpenAI-compatible provider, supporting both **single-model** and **multi-model** service plans with streaming enabled by default.
## Prerequisites
- A VMware Tanzu Platform (TAS) foundation with GenAI tile installed and configured
- Access to a CF org/space where the `genai` service is available in the marketplace
- The CF CLI (`cf`) installed and authenticated (`cf login`)
- goose v1.28.0 or later
## Step 1: Check Available Plans
First, verify the `genai` service is available in your marketplace and review the available plans:
```sh
cf marketplace -e genai
```
You will see output similar to:
```
broker: genai-service
plan description free or paid
tanzu-Qwen3-Coder-30B-A3B-vllm-v1 Access to: Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8. free
tanzu-gpt-oss-120b-vllm-v1 Access to: openai/gpt-oss-120b. free
tanzu-all-models Access to: Qwen3.5-122B, Qwen3-Coder-30B, gpt-oss... free
```
Each plan corresponds to a different model or set of models. **Single-model plans** give access to one model. **Multi-model plans** (e.g., `tanzu-all-models`) give access to multiple models behind a single endpoint.
## Step 2: Create a Service Instance
### Option A: Single-Model Plan
Create a service instance using a single-model plan:
```sh
cf create-service genai tanzu-Qwen3-Coder-30B-A3B-vllm-v1 my-qwen-coder --wait
```
### Option B: Multi-Model Plan
Create a service instance using the multi-model plan:
```sh
cf create-service genai tanzu-all-models my-all-models --wait
```
Verify the instance was created:
```sh
cf services
```
## Step 3: Create a Service Key
Create a service key to generate API credentials:
```sh
cf create-service-key my-qwen-coder my-goose-key --wait
```
Then retrieve the credentials:
```sh
cf service-key my-qwen-coder my-goose-key
```
### Single-Model Plan Output
For a single-model plan, the output includes model metadata at the top level:
```json
{
"credentials": {
"api_base": "https://genai-proxy.sys.example.com/tanzu-my-model-abc1234/openai",
"api_key": "eyJhbGciOi...",
"endpoint": {
"api_base": "https://genai-proxy.sys.example.com/tanzu-my-model-abc1234",
"api_key": "eyJhbGciOi...",
"config_url": "https://genai-proxy.sys.example.com/tanzu-my-model-abc1234/config/v1/endpoint",
"name": "tanzu-my-model-abc1234"
},
"model_capabilities": ["chat", "tools"],
"model_name": "Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8",
"wire_format": "openai"
}
}
```
### Multi-Model Plan Output
For a multi-model plan, the output only contains the endpoint object:
```json
{
"credentials": {
"endpoint": {
"api_base": "https://genai-proxy.sys.example.com/tanzu-all-models-abc1234",
"api_key": "eyJhbGciOi...",
"config_url": "https://genai-proxy.sys.example.com/tanzu-all-models-abc1234/config/v1/endpoint",
"name": "tanzu-all-models-abc1234"
}
}
}
```
## Step 4: Identify Your Endpoint and API Key
From the service key output, you need two values from the **`credentials.endpoint`** object:
| Value | JSON Path | Example |
|-------|-----------|---------|
| **Endpoint URL** | `credentials.endpoint.api_base` | `https://genai-proxy.sys.example.com/tanzu-my-model-abc1234` |
| **API Key** | `credentials.endpoint.api_key` | `eyJhbGciOi...` (JWT token) |
:::warning Use `credentials.endpoint.api_base`, not `credentials.api_base`
Single-model plans include a top-level `credentials.api_base` field that has an `/openai` suffix. **Do not use this value.** Always use `credentials.endpoint.api_base` (without `/openai`), because goose automatically appends the correct path.
Using the wrong value would produce a double-path URL like `.../openai/openai/v1/chat/completions`.
:::
## Step 5: Configure goose
<Tabs groupId="interface">
<TabItem value="ui" label="goose Desktop" default>
1. Open goose Desktop
2. Click the sidebar button, then **Settings** > **Models** > **Configure providers**
3. Find **VMware Tanzu Platform** in the provider list and click **Configure**
4. Enter your values:
- **TANZU_AI_ENDPOINT**: Paste the `credentials.endpoint.api_base` URL
- **TANZU_AI_API_KEY**: Paste the `credentials.endpoint.api_key` JWT token
5. Click **Submit**
6. Select a model from the dynamically fetched list
</TabItem>
<TabItem value="cli" label="goose CLI">
### Option 1: Using `goose configure`
```sh
goose configure
```
1. Select **Configure Providers**
2. Choose **VMware Tanzu Platform** from the list
3. Enter your `TANZU_AI_ENDPOINT` when prompted
4. Enter your `TANZU_AI_API_KEY` when prompted
5. Select a model from the fetched list
### Option 2: Using environment variables
Set the following environment variables before launching goose:
```sh
export TANZU_AI_ENDPOINT="https://genai-proxy.sys.example.com/tanzu-my-model-abc1234"
export TANZU_AI_API_KEY="eyJhbGciOi..."
```
Then start goose:
```sh
goose session
```
:::tip
Add these exports to your shell profile (`~/.bashrc`, `~/.zshrc`, etc.) to persist them across sessions.
:::
</TabItem>
</Tabs>
## Step 6: Select a Model
goose dynamically fetches available models from your Tanzu endpoint. After configuring the provider:
- **Single-model plan**: The one available model will be listed (e.g., `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8`)
- **Multi-model plan**: All models on the plan will be listed, and you can switch between them
To change models later, use **Settings** > **Models** > **Switch models** in Desktop, or run `goose configure` in the CLI.
:::note
Embedding-only models (e.g., `nomic-ai/nomic-embed-text-v2-moe`) will appear in the model list but cannot be used as a chat model.
:::
## Troubleshooting
### "Could not contact provider" / 401 Unauthorized on models endpoint
This means the API key is not being sent correctly. Common causes:
1. **Environment variables not set**: If using goose Desktop, env vars from your shell may not be inherited. Use the Settings UI to configure the provider instead.
2. **Wrong `api_base`**: Make sure you used `credentials.endpoint.api_base` (without `/openai`), not `credentials.api_base`.
3. **Expired API key**: Tanzu API keys are JWT tokens that may expire. Generate a new service key with `cf create-service-key`.
### Verify your endpoint manually
You can test connectivity with curl:
```sh
# Test model discovery
curl -H "Authorization: Bearer $TANZU_AI_API_KEY" \
"$TANZU_AI_ENDPOINT/openai/v1/models"
# Test chat completions
curl -X POST "$TANZU_AI_ENDPOINT/openai/v1/chat/completions" \
-H "Authorization: Bearer $TANZU_AI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"YOUR_MODEL_NAME","messages":[{"role":"user","content":"hello"}]}'
```
### Streaming
Streaming is enabled by default. If your endpoint does not support streaming, you can disable it by unchecking the **Streaming** checkbox in the provider configuration UI, or by setting the `TANZU_AI_STREAMING` environment variable to `false`.
### Model not found
If the model you selected returns an error, verify available models on your plan:
```sh
curl -H "Authorization: Bearer $TANZU_AI_API_KEY" \
"$TANZU_AI_ENDPOINT/openai/v1/models"
```
Ensure the model name matches exactly (including the prefix, e.g., `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8`).
### Cleaning up
To remove a service instance and its keys:
```sh
cf delete-service-key my-qwen-coder my-goose-key -f
cf delete-service my-qwen-coder -f
```
documentation/docs/guides/tanzu-cli-testing-guide.md
+149
View File
@@ -0,0 +1,149 @@
# VMware Tanzu Platform - CLI Testing Guide
## Prerequisites
- goose CLI built from the `feat/tanzu-ai-provider` branch
- A Tanzu AI Services endpoint and API key (single-model or multi-model plan)
## Locate the CLI Binary
**macOS:**
```bash
# If built from source:
export GOOSE_CLI=~/claude/goose-fork/target/release/goose
# Verify:
$GOOSE_CLI --version
```
**Linux:**
```bash
# If installed via .deb:
export GOOSE_CLI=/usr/bin/goose
# If built from source:
export GOOSE_CLI=~/goose-fork/target/release/goose
# Verify:
$GOOSE_CLI --version
```
## Test 1: Configure VMware Tanzu Platform Provider
```bash
goose configure
```
1. Select **Configure Providers**
2. Scroll to / search for **VMware Tanzu Platform**
3. When prompted for **TANZU_AI_ENDPOINT**, enter your endpoint URL:
- Single-model: `https://genai-proxy.sys.example.com/tanzu-my-model-abc1234`
- Multi-model: `https://genai-proxy.sys.example.com/tanzu-all-models-abc1234`
4. When prompted for **TANZU_AI_API_KEY**, paste the JWT token from your service key
5. Select a model from the dynamically fetched list
**Expected:** Models are fetched from the endpoint and displayed for selection.
## Test 2: Start a Session (Single-Model Plan)
```bash
export TANZU_AI_ENDPOINT="https://genai-proxy.sys.tas-tdc.kuhn-labs.com/tanzu-Qwen3-Coder-30B-A3B-vllm-v1-f3b0d18"
export TANZU_AI_API_KEY="<your-jwt-token>"
goose session
```
Type a simple prompt:
```
> What is 2 + 2?
```
**Expected:** The model responds with an answer. If streaming is enabled, tokens appear incrementally.
## Test 3: Start a Session (Multi-Model Plan)
```bash
export TANZU_AI_ENDPOINT="https://genai-proxy.sys.tas-tdc.kuhn-labs.com/tanzu-all-models-a8a9e22"
export TANZU_AI_API_KEY="<your-jwt-token>"
goose session
```
**Expected:** Session starts with whichever model was selected during `goose configure`.
## Test 4: Verify Streaming
With streaming enabled (`supports_streaming: true`), responses should appear token-by-token rather than all at once.
```
> Write a short poem about clouds
```
**Expected:** Text streams in progressively, not appearing all at once after a delay.
## Test 5: Verify Dynamic Model Fetching
```bash
goose configure
```
Select **Configure Providers** > **VMware Tanzu Platform**.
**Expected for single-model plan:** One model appears (e.g., `Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8`)
**Expected for multi-model plan:** Multiple models appear (e.g., `Qwen3.5-122B`, `Qwen3-Coder-30B`, `gpt-oss-120b`)
## Test 6: Verify Error Messages
### Missing API Key
```bash
unset TANZU_AI_API_KEY
goose session
```
**Expected:** Clear error message: "Required API key TANZU_AI_API_KEY is not set."
### Missing Endpoint
```bash
unset TANZU_AI_ENDPOINT
goose session
```
**Expected:** Clear error message about TANZU_AI_ENDPOINT not being set.
### Wrong Endpoint
```bash
export TANZU_AI_ENDPOINT="https://genai-proxy.sys.example.com/nonexistent"
export TANZU_AI_API_KEY="invalid-key"
goose session
```
**Expected:** Connection or authentication error, not a crash.
## Test 7: Switch Between Plans
1. Configure with multi-model endpoint, select a model, start a session, verify it works
2. Run `goose configure` again
3. Change TANZU_AI_ENDPOINT to the single-model endpoint
4. Select the single model
5. Start a new session, verify it works
**Expected:** Both plans work without needing to restart goose.
## Quick Curl Verification
Before testing with goose, you can verify endpoints directly:
```bash
# Test models endpoint
curl -s -H "Authorization: Bearer $TANZU_AI_API_KEY" \
"$TANZU_AI_ENDPOINT/openai/v1/models" | python3 -m json.tool
# Test chat completions
curl -s -X POST "$TANZU_AI_ENDPOINT/openai/v1/chat/completions" \
-H "Authorization: Bearer $TANZU_AI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8","messages":[{"role":"user","content":"hello"}],"max_tokens":10}'
# Test streaming
curl -s -N -X POST "$TANZU_AI_ENDPOINT/openai/v1/chat/completions" \
-H "Authorization: Bearer $TANZU_AI_API_KEY" \
-H "Content-Type: application/json" \
-d '{"model":"Qwen/Qwen3-Coder-30B-A3B-Instruct-FP8","messages":[{"role":"user","content":"hello"}],"max_tokens":10,"stream":true}'
```