The fix: add one line to the robots.txt file at the root of your domain:
Sitemap: https://example.com/sitemap.xml
Use the full URL, deploy, and re-run your audit. That clears the "sitemap.xml not indicated in robots.txt" warning.
The rest of this guide covers where that warning comes from, the syntax that trips people up, and the robots.txt and sitemap conflicts that do real damage to indexing.
What the warning means
"Sitemap.xml not indicated in robots.txt" is a notice from SEO audit tools such as Semrush's Site Audit. It means your robots.txt has no Sitemap: line. Google Search Console doesn't report it, and on its own it won't stop pages being indexed.
It's still worth fixing. Crawlers read robots.txt before crawling a site, so a Sitemap: line there is the simplest way to tell every search engine, not just Google, where your list of URLs lives. Google can also find a sitemap you submit in Search Console. Bing and smaller crawlers often rely on the robots.txt line.
robots.txt vs sitemap.xml: what each one does
The two files do opposite jobs, which is why they need to agree.
| robots.txt | sitemap.xml | |
|---|---|---|
| Job | Says which paths crawlers may or may not fetch | Lists the URLs you want discovered and indexed |
| Location | Always /robots.txt at the root of the host |
Anywhere, as long as it's declared or submitted |
| Typical contents | User-agent, Disallow, Allow, Sitemap lines |
A <urlset> of page URLs, or a <sitemapindex> of other sitemaps |
| Does it control indexing? | No. It controls crawling. A blocked page can still be indexed from links, without its content | No. It's a hint about which URLs matter |
To keep a page out of Google, use a noindex robots meta tag and don't block the page in robots.txt. If the page is blocked, Google never fetches it and never sees the noindex.
How to add your sitemap to robots.txt
A typical robots.txt with a sitemap line:
User-agent: *
Disallow: /admin/
Disallow: /dashboard/
Sitemap: https://example.com/sitemap.xml
On Next.js with the App Router, app/robots.js (or robots.ts) generates this file. Return a sitemap field there instead of editing a static file.
Syntax rules
- Use an absolute URL.
Sitemap: /sitemap.xmlis not valid. Includehttps://and the domain. - Use your canonical host and protocol. If your site lives at
https://example.com, don't point tohttp://orwww.. A sitemap URL that redirects adds a step that some crawlers skip. - Position doesn't matter. The
Sitemap:line isn't tied to anyUser-agentgroup, so it can go anywhere. The convention is the end of the file. - Case. Google treats the field name as case-insensitive, but the URL itself is case-sensitive. Write
Sitemap:with a capital S to be safe with every crawler. - One URL per line. Add a separate
Sitemap:line for each sitemap.
Multiple sitemaps or a sitemap index
If your site has a sitemap index file that points to other sitemaps, list just the index:
Sitemap: https://example.com/sitemap-index.xml
Or list each sitemap on its own line:
Sitemap: https://example.com/sitemap-posts.xml
Sitemap: https://example.com/sitemap-products.xml
Robots.txt and sitemap conflicts that hurt indexing
The missing Sitemap: line is the mild problem. These are the ones that keep pages out of Google:
- Sitemap URLs blocked by robots.txt. Your sitemap asks Google to index
/pricing, and robots.txt saysDisallow: /pricing. Search Console flags these as blocked by robots.txt. Decide which file is wrong and fix that one. - A leftover
Disallow: /. Staging sites often block everything. If that robots.txt ships to production, crawling stops across the whole site. - Blocking pages you want removed. If you add
noindexto a page and also disallow it, Google can't see thenoindex, and the URL can stay in results. - Blocking CSS or JavaScript. If Google can't fetch the files a page needs to render, it may misread the page. Leave
/_next/,/static/and similar asset paths crawlable. - Sitemap listing URLs that redirect or 404. Only list final, canonical URLs that return
200. A sitemap full of redirects tells Google the file isn't maintained.
Check your setup
After deploying:
- Open
https://yourdomain.com/robots.txtin a browser and confirm theSitemap:line is there with the right URL. - Run the Google Indexing Checker. It fetches your robots.txt, lists its
Sitemap:lines, flags disallow rules that apply to all crawlers, and checks whether/sitemap.xmlis reachable and declared. - Run the Sitemap Validator to confirm the sitemap is valid XML, see how many URLs it lists, and check
lastmodand content type. - In Google Search Console, submit the sitemap under Sitemaps and watch its status. Our guide on how to check a website's sitemap explains each status.
Both tools are free and need no sign-up.
Next: AI crawlers
The same robots.txt also decides whether AI crawlers such as GPTBot, ClaudeBot and PerplexityBot can read your site. See our guide to llms.txt and AI crawlers for what to allow and why.