fix(vision): stop read_image from poisoning text-provider sessions
Image turns already get a vision-model description injected into the prompt, but the agent kept calling read_image to "confirm" the pictures. Those tool results carry base64 image parts that Goose persists, so every later turn against the text-only chat provider failed with `unknown variant image_url` before the agent could write the page. WeChat page requests therefore fell through to the fail-closed delivery message. Drop read_image for the turn whenever a vision model handles the images, say so explicitly in the injected prompt, and teach the poison scan to recognise tool image parts so already-polluted sessions rotate instead of failing again. Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
@@ -186,6 +186,35 @@ export function messageContentHasImageUrl(content) {
|
||||
return content.some((item) => item?.type === 'image_url' && item?.image_url?.url);
|
||||
}
|
||||
|
||||
function toolResultParts(item) {
|
||||
const result = item?.toolResult ?? item?.tool_result ?? item?.toolResponse ?? item?.tool_response;
|
||||
const value = result?.value ?? result;
|
||||
if (Array.isArray(value)) return value;
|
||||
if (Array.isArray(value?.content)) return value.content;
|
||||
return [];
|
||||
}
|
||||
|
||||
/**
|
||||
* REGRESSION GUARD: vision-turn-read-image-isolation
|
||||
* `read_image` answers with base64 image parts nested in the tool response.
|
||||
* Those parts never appear as `image_url`, but once Goose persists them a
|
||||
* text-only chat provider rejects the whole history with
|
||||
* `unknown variant image_url, expected text`, so they poison the session too.
|
||||
*/
|
||||
export function messageContentHasToolImagePart(content) {
|
||||
if (!Array.isArray(content)) return false;
|
||||
return content.some((item) => {
|
||||
const type = String(item?.type ?? '');
|
||||
if (type !== 'toolResponse' && type !== 'tool_response') return false;
|
||||
return toolResultParts(item).some((part) => String(part?.type ?? '') === 'image');
|
||||
});
|
||||
}
|
||||
|
||||
export function conversationHasToolImageContent(conversation) {
|
||||
if (!Array.isArray(conversation)) return false;
|
||||
return conversation.some((message) => messageContentHasToolImagePart(message?.content));
|
||||
}
|
||||
|
||||
/**
|
||||
* Any persisted image_url part will break DeepSeek / other text-only providers.
|
||||
* User metadata.imageUrls alone is not enough — Goose may have expanded them into
|
||||
|
||||
Reference in New Issue
Block a user