fix(vision): stop read_image from poisoning text-provider sessions

Image turns already get a vision-model description injected into the prompt,
but the agent kept calling read_image to "confirm" the pictures. Those tool
results carry base64 image parts that Goose persists, so every later turn
against the text-only chat provider failed with `unknown variant image_url`
before the agent could write the page. WeChat page requests therefore fell
through to the fail-closed delivery message.

Drop read_image for the turn whenever a vision model handles the images, say
so explicitly in the injected prompt, and teach the poison scan to recognise
tool image parts so already-polluted sessions rotate instead of failing again.

Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
john
2026-08-22 10:40:12 +08:00
parent 4d12ea438b
commit ef4ce12bbf
9 changed files with 457 additions and 6 deletions
+29
View File
@@ -186,6 +186,35 @@ export function messageContentHasImageUrl(content) {
return content.some((item) => item?.type === 'image_url' && item?.image_url?.url);
}
function toolResultParts(item) {
const result = item?.toolResult ?? item?.tool_result ?? item?.toolResponse ?? item?.tool_response;
const value = result?.value ?? result;
if (Array.isArray(value)) return value;
if (Array.isArray(value?.content)) return value.content;
return [];
}
/**
* REGRESSION GUARD: vision-turn-read-image-isolation
* `read_image` answers with base64 image parts nested in the tool response.
* Those parts never appear as `image_url`, but once Goose persists them a
* text-only chat provider rejects the whole history with
* `unknown variant image_url, expected text`, so they poison the session too.
*/
export function messageContentHasToolImagePart(content) {
if (!Array.isArray(content)) return false;
return content.some((item) => {
const type = String(item?.type ?? '');
if (type !== 'toolResponse' && type !== 'tool_response') return false;
return toolResultParts(item).some((part) => String(part?.type ?? '') === 'image');
});
}
export function conversationHasToolImageContent(conversation) {
if (!Array.isArray(conversation)) return false;
return conversation.some((message) => messageContentHasToolImagePart(message?.content));
}
/**
* Any persisted image_url part will break DeepSeek / other text-only providers.
* User metadata.imageUrls alone is not enough — Goose may have expanded them into