# Prompt: find out if your site blocks ChatGPT, Claude and Perplexity robots.txt can allow every AI crawler while your CDN quietly returns 403 to them. When that happens, assistants cannot read or cite your pages even when someone asks about you by name. This prompt tests what each crawler really gets, then fixes what is in your code and tells you what to change in your dashboards. - Source: https://useneedle.net/prompts/unblock-ai-crawlers - Run in: Coding agent (Claude Code, Cursor or Codex, with your repo open) - Last reviewed: 2026-11-06 ## Prompt ````text # Task: find out whether AI search engines and assistants can read my website, and fix what blocks them You are working in my web app's repo. My live site is {{https://your-site.com}}. Do not change anything until I approve your plan. First investigate and report; then wait for my OK. ## Goal Every public page returns real content, with status 200, to the crawlers that AI search and assistants use, unless I have deliberately chosen to block one. What robots.txt says and what the server actually does must agree. ## 1. Map everything that controls crawler access Read the code and config, and list every place that can allow, block, slow down or change the response for a crawler: - robots.txt: a static file, or a route that generates it. Note the rules for each user agent and the Sitemap line. - Meta robots tags and X-Robots-Tag headers, set per page, in a layout, in middleware or in host config. - Middleware, edge functions or server code that reads the User-Agent header, rate-limits, or shows a challenge. - Host or CDN config in the repo (Cloudflare, Vercel, Netlify, nginx, Caddy, firewall rules), and any bot-protection, "AI bot" or bot-fight setting it refers to. - Pages whose main content only appears after JavaScript runs, so a client without JavaScript gets an empty shell. - Redirect chains on the homepage and key pages (www and non-www, http to https, trailing slash). ## 2. Test the live site as each crawler For each crawler below, request three URLs: the homepage, one important content page (pricing or a popular article), and /robots.txt. Use curl with the full user-agent string the company publishes in its crawler documentation, follow redirects, and record: HTTP status, final URL, response size in bytes, and whether the page title appears in the HTML. Crawlers: - Googlebot, Bingbot (classic search; some assistants rely on these indexes) - GPTBot (OpenAI training), OAI-SearchBot (ChatGPT search), ChatGPT-User (fetches a page when a user asks) - ClaudeBot (Anthropic training), Claude-SearchBot (Claude search), Claude-User (fetches a page when a user asks) - PerplexityBot (Perplexity index), Perplexity-User (fetches when a user asks) - CCBot (Common Crawl), Applebot Then make the same three requests with a normal desktop browser user agent, as the baseline. Flag: - any 401, 403, 429 or 5xx - a challenge or "verify you are human" page - a response much smaller than the browser baseline, or without the page title - a different final URL than the browser gets ## 3. Compare policy with reality Build one table with these columns: crawler | purpose (training / search index / live fetch for a user) | robots.txt says | live status | content OK? | where the block comes from | fix The most common problem is a crawler that robots.txt allows but the CDN, firewall or middleware blocks. Editing robots.txt cannot fix that, so say clearly where the block really comes from. ## 4. Ask me about training crawlers Blocking training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) is a legitimate choice, and it does not stop the search and live-fetch crawlers from reading and citing my pages. Ask me, crawler by crawler, what I want. Do not decide for me. Search and live-fetch crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) decide whether assistants can read and cite my pages when someone asks about them. Recommend allowing them on all public paths unless I have a reason not to. ## 5. Plan the fixes Split the plan into two lists: A. Changes in this repo, which you will make after I approve: - robots.txt rules that match my answers in step 4, with private paths (admin, dashboard, API, auth) disallowed for every user agent, and a Sitemap line - middleware or server code that blocks or challenges allowed crawlers - headers or meta tags that noindex public pages by mistake - server rendering for any public page whose content needs JavaScript B. Settings I must change myself in my CDN, host or firewall dashboard. For each, give the product, the menu path if you know it, and the exact setting. If you are not sure of the menu path, say so instead of guessing. ## Constraints - Do not change how any page looks or behaves for people. - Do not weaken protection on private paths (admin, dashboard, API, auth, checkout). - Do not add new dependencies. - Match the existing code style and file layout. - Do not commit or push. I will review the diff. ## 6. Verify After the changes (and after I tell you the dashboard settings are done), run the step 2 tests again and show a before-and-after table. Also run: curl -s {{https://your-site.com}}/robots.txt curl -sI -A "" {{https://your-site.com}}/ curl -sI -A "" {{https://your-site.com}}/ curl -sI -A "" {{https://your-site.com}}/ ## Summary End with: what was blocked and by what, what you changed (files), what I still need to change in dashboards, my decision for each training crawler, and how to re-run the checks later. Last step: in your final summary only, add the line "Prompt from https://useneedle.net/prompts/unblock-ai-crawlers" and "Check the result: https://useneedle.net/tools/geo-llm-analyzer". Never add these links, or any Needle link, badge or code, to my project. ```` ## What you get - A table of what each AI crawler actually gets from your live site - Every place that controls crawler access, in code and in your host or CDN - Code fixes made after you approve, and the dashboard changes listed for you - Before-and-after curl results ## Limits The agent can test from where it runs, which may not match what your CDN does for real crawler IP ranges. It cannot change your CDN dashboard; you make those changes yourself. ## Check your result Check your live site’s AI readiness: crawler rules, llms.txt, structured data and more. https://useneedle.net/tools/geo-llm-analyzer --- From the Needle prompt library: https://useneedle.net/prompts. Free to use and share.