robots.txt and sitemap.xml: robots.txt Best Practices and Sitemap Rules
robots.txt tells search engine crawlers where not to go on your site, while sitemap.xml lists the pages you want them to find and index. One mistake in them can remove a site from Google or flood the index with junk URLs. Below are robots.txt best practices: the syntax, what to block and what never to block, how to handle AI crawlers, the rules for sitemap.xml and how to check everything in Search Console.
Contents
- robots.txt vs sitemap.xml: what is the difference
- robots.txt syntax: User-agent, Disallow, Allow, Sitemap
- robots.txt blocks crawling, not indexing
- What to block in robots.txt and what never to block
- AI crawlers in robots.txt: block or allow
- sitemap.xml rules: what belongs in a sitemap
- How to set up robots.txt and sitemap.xml: 5 steps
- Common robots.txt and sitemap.xml mistakes
- Checklist: robots.txt and sitemap.xml
- Frequently asked questions
- Summary
robots.txt vs sitemap.xml: what is the difference
In short: robots.txt limits crawling, sitemap.xml helps direct it.
| robots.txt | sitemap.xml | |
|---|---|---|
| Job | Tell crawlers which sections not to crawl | List the pages that should be found and indexed |
| Location | Only at the host root: https://site.com/robots.txt | Anywhere, usually https://site.com/sitemap.xml |
| Format | Plain text, UTF-8 | XML, UTF-8, full absolute URLs |
| Google limits | Reads the first 500 KiB of the file | Up to 50,000 URLs and 50 MB uncompressed per file |
| Required? | No, without it the whole site is crawlable | No, but strongly recommended for large and new sites |
Note: robots.txt applies only to the host, protocol and port where it lives. A file on site.com does not control the subdomain shop.site.com - that one needs its own.
robots.txt syntax: User-agent, Disallow, Allow, Sitemap
The file consists of groups. Each group starts with a User-agent line (which crawler the rules are for), followed by Disallow and Allow lines. The Sitemap directive stands on its own and applies to all crawlers.
| Directive | What it does | Example |
|---|---|---|
| User-agent | Names the crawler the group applies to. * means all crawlers | User-agent: Googlebot |
| Disallow | Blocks crawling of a path and everything that starts with it | Disallow: /admin/ |
| Allow | Allows a path inside a blocked section | Allow: /admin/public/ |
| Sitemap | Full URL of the sitemap, multiple lines allowed | Sitemap: https://site.com/sitemap.xml |
| * | Any sequence of characters | Disallow: /*?sort= |
| $ | End of the URL | Disallow: /*.pdf$ |
Key rules:
- Paths are case-sensitive. Disallow: /Admin/ does not block /admin/;
- The longest rule wins. If both Disallow and Allow match, Google applies the one with the longer path. If they are equal in length, the least restrictive one wins, which is Allow;
- A crawler follows one group. Googlebot looks for the group with the most specific User-agent. If there is a group for Googlebot, the general * group no longer applies to it;
- Google ignores Crawl-delay. Other search engines may respect it.
Example robots.txt for an online store
A typical file:
| Line | Meaning |
|---|---|
| User-agent: * | Rules for all crawlers |
| Disallow: /admin/ | Admin area |
| Disallow: /cart/ | Shopping cart |
| Disallow: /checkout/ | Checkout |
| Disallow: /search | Internal site search |
| Disallow: /*?sort= | Catalog sorting |
| Disallow: /*filter= | Filter combinations you do not need in the index |
| Sitemap: https://site.com/sitemap.xml | Where the sitemap lives |
Before blocking parameters, check whether any of them are filter pages you deliberately want to rank, for example "men's Nike sneakers". Those pages should get clean URLs and stay open.
robots.txt blocks crawling, not indexing
Disallow stops a crawler from reading a page, but it does not stop Google from showing that URL in search. If other pages link to a blocked URL, it can get indexed without a description - in Search Console this shows up as "Indexed, though blocked by robots.txt".
To remove a page from search, use a meta robots noindex tag or the X-Robots-Tag: noindex HTTP header. And do not block that page in robots.txt: if Googlebot cannot crawl it, it will never see the noindex.
Google has not supported a noindex directive inside robots.txt since 2019. For duplicates that should pass their signals to a main page, the right tool is canonical, not a crawl block.
What to block in robots.txt and what never to block
Block what has no search value and generates lots of URLs: system sections, user accounts, internal search results, sorting and endless filter combinations. Google itself recommends doing this for internal search and unneeded faceted filters.
- Block: /admin/, /wp-admin/ (with Allow for /wp-admin/admin-ajax.php on WordPress), cart, checkout, user account, internal search, sorting parameters, UTM and other tracking parameters, staging sections;
- Never block: CSS, JavaScript and images needed to render the page. Google renders pages like a browser, and without these files it may see the page as empty or not mobile-friendly;
- Never leave after launch: Disallow: / from the staging server. This is the classic migration mistake - the whole site is blocked and nobody notices for weeks. More on launches in SEO for a new website.
AI crawlers in robots.txt: block or allow
robots.txt now also decides your site's presence in ChatGPT, Claude, Perplexity and Gemini. Each company runs several crawlers for different jobs, and you can block them selectively.
| User-agent | Owner | Purpose |
|---|---|---|
| GPTBot | OpenAI | Collecting data for model training |
| OAI-SearchBot | OpenAI | ChatGPT search: showing and citing sites in answers |
| ClaudeBot | Anthropic | Collecting data for training Claude models |
| PerplexityBot | Perplexity | Indexing for Perplexity search answers |
| Google-Extended | Not a separate crawler but a token: whether content may be used to train and ground Gemini. Has no effect on Google Search |
Blocking GPTBot or ClaudeBot keeps your content out of model training, but also lowers the chance that a model knows about your brand. Blocking OAI-SearchBot or PerplexityBot removes you from ChatGPT and Perplexity search answers, which means losing a traffic channel. For most commercial sites that need customers, AI crawlers are worth keeping open.
Keep in mind: robots.txt is an agreement, not protection, and agents triggered by a user (for example, when someone asks ChatGPT to open a page) may not follow these rules, according to the companies' own documentation. Also, AI bots are often blocked not by robots.txt but by hosting or Cloudflare bot protection. How to check your visibility in AI assistants is covered in how to get mentioned in ChatGPT.
I will check your robots.txt and sitemap.xml
Send me your site URL - I will check whether anything important is blocked, whether junk is getting indexed and whether your sitemap is built correctly. The express audit is free.
Message me on Telegramsitemap.xml rules: what belongs in a sitemap
A sitemap is a list of pages you want to see in search. For Google it is a hint, not an order.
- Only canonical URLs returning 200. No redirects, 404s, noindex pages or duplicates whose canonical points elsewhere;
- Full absolute URLs with the same protocol and domain as the main version of the site: https://site.com/page/, not /page/;
- Up to 50,000 URLs and 50 MB uncompressed per file. If you have more, split them into several files;
- A sitemap index file combines several sitemaps, for example separate ones for categories, products and articles. That index is what you submit to Search Console;
- Accurate lastmod. Google uses the last modified date if it is consistently accurate. Update it when the page content changes, not every time the file is generated;
- Google ignores priority and changefreq. You can leave them out.
How to set up robots.txt and sitemap.xml: 5 steps
List what needs to be blocked
Crawl the site with Screaming Frog or Sitebulb and find out which URL types it generates. For each type, decide: index it, block crawling or add noindex.
Write robots.txt
Start with a User-agent: * group, add Disallow lines for system sections and parameters, separate groups for AI crawlers if you decide to block them, and a Sitemap line with the full URL. Save it at the site root.
Set up sitemap.xml generation
Turn on the automatic sitemap in your CMS or plugin and make sure it includes only canonical pages returning 200. For a large site, split sitemaps by page type and combine them with an index file.
Submit the sitemap in Search Console
In Google Search Console, open Indexing - Sitemaps, paste the file URL and click Submit. The "Success" status means Google has read the file. How to use the other reports is covered in the article on Google Search Console.
Check the result
Search Console Settings include a robots.txt report: which file Google fetched, when, and whether there are errors. The URL Inspection tool shows whether crawling of a specific page is allowed. Google caches robots.txt for up to 24 hours, so changes do not apply instantly.
Common robots.txt and sitemap.xml mistakes
- Disallow: / after launch. The staging block was carried over to the live site, and traffic drops to zero;
- Blocked CSS and JS. Google sees a "broken" page;
- Disallow instead of noindex. Pages you wanted out of search stay indexed without a description;
- Junk in the sitemap. Redirects, 404s, noindex pages and non-canonical duplicates - Google trusts such a sitemap less;
- robots.txt returns a 5xx error. Google may temporarily stop crawling the site while the file is unavailable. A 404, on the other hand, means "no restrictions".
Checklist: robots.txt and sitemap.xml
- robots.txt is at the root of every host and returns 200
- No Disallow: / on the live site
- CSS, JS and images are crawlable
- Admin, cart, checkout, internal search and unneeded parameters are blocked
- Pages with noindex are not blocked in robots.txt
- The decision on GPTBot, OAI-SearchBot, ClaudeBot, PerplexityBot and Google-Extended is made deliberately
- robots.txt has a Sitemap line with the full URL
- sitemap.xml contains only canonical 200 URLs without noindex
- Each sitemap file is under 50,000 URLs and 50 MB, large sites use an index file
- lastmod reflects the real date the page changed
- The sitemap is submitted in Search Console with the "Success" status
Frequently asked questions
Does every website need a robots.txt file?
No. Without robots.txt, Google crawls the whole site. But online stores and sites with filters almost always need one, so crawlers do not waste time on thousands of technical URLs.
How do I keep a page out of the index: robots.txt or noindex?
Use noindex. robots.txt only blocks crawling, and a blocked page can still be indexed without a description. A page with noindex must stay crawlable in robots.txt, otherwise Google will not see the directive.
Does a small website need a sitemap.xml?
For a site with a few dozen pages and good internal linking, a sitemap is not critical, but it does no harm. For new and large sites, sitemap.xml is strongly recommended.
What are the main robots.txt best practices for WordPress?
WordPress generates a virtual robots.txt automatically. You can change it through an SEO plugin (Yoast, Rank Math) or by uploading your own file to the site root. Typically you block /wp-admin/ with Allow for /wp-admin/admin-ajax.php, internal search and unneeded parameters.
Does blocking AI crawlers in robots.txt hurt SEO?
Blocking GPTBot, ClaudeBot or Google-Extended does not affect Google rankings. But blocking OAI-SearchBot and PerplexityBot removes the site from ChatGPT and Perplexity search answers, so for a commercial site it means losing potential customers.
Summary
robots.txt controls where crawlers go, sitemap.xml controls what they should find, and noindex and canonical control what ends up in the index. Block system sections, search and unneeded filters, but never CSS, JS or the whole site. Keep only canonical 200 pages in your sitemap and check the Search Console reports every month.
I will set up your technical SEO
robots.txt, sitemap, indexing, duplicates and speed - as part of monthly SEO from $450/month. We start with a free express audit.
Send a request