docs: adding AI-friendly features (#5918)

Co-authored-by: Copilot <175728472+Copilot@users.noreply.github.com>
2025-12-01 04:44:20 +01:00
parent ccf939d215
commit fa7ce8ec94
4 changed files with 145 additions and 1 deletions
@@ -0,0 +1,33 @@
+# goose Documentation
+# https://block.github.io/goose/
+
+User-agent: *
+Allow: /
+
+# Sitemap
+Sitemap: https://block.github.io/goose/sitemap.xml
+
+# LLM context file available at /llms.txt
+
+
+# AI crawlers are welcome
+User-agent: GPTBot
+Allow: /
+
+User-agent: Claude-Web
+Allow: /
+
+User-agent: Anthropic-AI
+Allow: /
+
+User-agent: Google-Extended
+Allow: /
+
+User-agent: PerplexityBot
+Allow: /
+
+User-agent: Bytespider
+Allow: /
+
+User-agent: CCBot
+Allow: /