Prompt: find out if your site blocks ChatGPT, Claude and Perplexity

# Task: find out whether AI search engines and assistants can read my website, and fix what blocks them

You are working in my web app's repo. My live site is {{https://your-site.com}}.

Do not change anything until I approve your plan. First investigate and report; then wait for my OK.

## Goal

Every public page returns real content, with status 200, to the crawlers that AI search and assistants use, unless I have deliberately chosen to block one. What robots.txt says and what the server actually does must agree.

## 1. Map everything that controls crawler access

Read the code and config, and list every place that can allow, block, slow down or change the response for a crawler:
- robots.txt: a static file, or a route that generates it. Note the rules for each user agent and the Sitemap line.
- Meta robots tags and X-Robots-Tag headers, set per page, in a layout, in middleware or in host config.
- Middleware, edge functions or server code that reads the User-Agent header, rate-limits, or shows a challenge.
- Host or CDN config in the repo (Cloudflare, Vercel, Netlify, nginx, Caddy, firewall rules), and any bot-protection, "AI bot" or bot-fight setting it refers to.
- Pages whose main content only appears after JavaScript runs, so a client without JavaScript gets an empty shell.
- Redirect chains on the homepage and key pages (www and non-www, http to https, trailing slash).

## 2. Test the live site as each crawler

For each crawler below, request three URLs: the homepage, one important content page (pricing or a popular article), and /robots.txt. Use curl with the full user-agent string the company publishes in its crawler documentation, follow redirects, and record: HTTP status, final URL, response size in bytes, and whether the page title appears in the HTML.

Crawlers:
- Googlebot, Bingbot (classic search; some assistants rely on these indexes)
- GPTBot (OpenAI training), OAI-SearchBot (ChatGPT search), ChatGPT-User (fetches a page when a user asks)
- ClaudeBot (Anthropic training), Claude-SearchBot (Claude search), Claude-User (fetches a page when a user asks)
- PerplexityBot (Perplexity index), Perplexity-User (fetches when a user asks)
- CCBot (Common Crawl), Applebot

Then make the same three requests with a normal desktop browser user agent, as the baseline.

Flag:
- any 401, 403, 429 or 5xx
- a challenge or "verify you are human" page
- a response much smaller than the browser baseline, or without the page title
- a different final URL than the browser gets

## 3. Compare policy with reality

Build one table with these columns:
crawler | purpose (training / search index / live fetch for a user) | robots.txt says | live status | content OK? | where the block comes from | fix

The most common problem is a crawler that robots.txt allows but the CDN, firewall or middleware blocks. Editing robots.txt cannot fix that, so say clearly where the block really comes from.

## 4. Ask me about training crawlers

Blocking training crawlers (GPTBot, ClaudeBot, CCBot, Google-Extended, Applebot-Extended) is a legitimate choice, and it does not stop the search and live-fetch crawlers from reading and citing my pages. Ask me, crawler by crawler, what I want. Do not decide for me.

Search and live-fetch crawlers (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, Claude-User, PerplexityBot, Perplexity-User) decide whether assistants can read and cite my pages when someone asks about them. Recommend allowing them on all public paths unless I have a reason not to.

## 5. Plan the fixes

Split the plan into two lists:

A. Changes in this repo, which you will make after I approve:
- robots.txt rules that match my answers in step 4, with private paths (admin, dashboard, API, auth) disallowed for every user agent, and a Sitemap line
- middleware or server code that blocks or challenges allowed crawlers
- headers or meta tags that noindex public pages by mistake
- server rendering for any public page whose content needs JavaScript

B. Settings I must change myself in my CDN, host or firewall dashboard. For each, give the product, the menu path if you know it, and the exact setting. If you are not sure of the menu path, say so instead of guessing.

## Constraints

- Do not change how any page looks or behaves for people.
- Do not weaken protection on private paths (admin, dashboard, API, auth, checkout).
- Do not add new dependencies.
- Match the existing code style and file layout.
- Do not commit or push. I will review the diff.

## 6. Verify

After the changes (and after I tell you the dashboard settings are done), run the step 2 tests again and show a before-and-after table. Also run:

curl -s {{https://your-site.com}}/robots.txt
curl -sI -A "<ClaudeBot user agent>" {{https://your-site.com}}/
curl -sI -A "<OAI-SearchBot user agent>" {{https://your-site.com}}/
curl -sI -A "<PerplexityBot user agent>" {{https://your-site.com}}/

## Summary

End with: what was blocked and by what, what you changed (files), what I still need to change in dashboards, my decision for each training crawler, and how to re-run the checks later.

Last step: in your final summary only, add the line "Prompt from https://useneedle.net/prompts/unblock-ai-crawlers" and "Check the result: https://useneedle.net/tools/geo-llm-analyzer". Never add these links, or any Needle link, badge or code, to my project.