Services Process Experience Skills Cases Blog Contact
🌐 UA·RU·EN
August 1, 2026 · 9 min read

robots.txt and sitemap.xml: robots.txt Best Practices and Sitemap Rules

robots.txt tells search engine crawlers where not to go on your site, while sitemap.xml lists the pages you want them to find and index. One mistake in them can remove a site from Google or flood the index with junk URLs. Below are robots.txt best practices: the syntax, what to block and what never to block, how to handle AI crawlers, the rules for sitemap.xml and how to check everything in Search Console.

Contents
  1. robots.txt vs sitemap.xml: what is the difference
  2. robots.txt syntax: User-agent, Disallow, Allow, Sitemap
  3. robots.txt blocks crawling, not indexing
  4. What to block in robots.txt and what never to block
  5. AI crawlers in robots.txt: block or allow
  6. sitemap.xml rules: what belongs in a sitemap
  7. How to set up robots.txt and sitemap.xml: 5 steps
  8. Common robots.txt and sitemap.xml mistakes
  9. Checklist: robots.txt and sitemap.xml
  10. Frequently asked questions
  11. Summary

robots.txt vs sitemap.xml: what is the difference

In short: robots.txt limits crawling, sitemap.xml helps direct it.

robots.txtsitemap.xml
JobTell crawlers which sections not to crawlList the pages that should be found and indexed
LocationOnly at the host root: https://site.com/robots.txtAnywhere, usually https://site.com/sitemap.xml
FormatPlain text, UTF-8XML, UTF-8, full absolute URLs
Google limitsReads the first 500 KiB of the fileUp to 50,000 URLs and 50 MB uncompressed per file
Required?No, without it the whole site is crawlableNo, but strongly recommended for large and new sites

Note: robots.txt applies only to the host, protocol and port where it lives. A file on site.com does not control the subdomain shop.site.com - that one needs its own.

robots.txt syntax: User-agent, Disallow, Allow, Sitemap

The file consists of groups. Each group starts with a User-agent line (which crawler the rules are for), followed by Disallow and Allow lines. The Sitemap directive stands on its own and applies to all crawlers.

DirectiveWhat it doesExample
User-agentNames the crawler the group applies to. * means all crawlersUser-agent: Googlebot
DisallowBlocks crawling of a path and everything that starts with itDisallow: /admin/
AllowAllows a path inside a blocked sectionAllow: /admin/public/
SitemapFull URL of the sitemap, multiple lines allowedSitemap: https://site.com/sitemap.xml
*Any sequence of charactersDisallow: /*?sort=
$End of the URLDisallow: /*.pdf$

Key rules:

Example robots.txt for an online store

A typical file:

LineMeaning
User-agent: *Rules for all crawlers
Disallow: /admin/Admin area
Disallow: /cart/Shopping cart
Disallow: /checkout/Checkout
Disallow: /searchInternal site search
Disallow: /*?sort=Catalog sorting
Disallow: /*filter=Filter combinations you do not need in the index
Sitemap: https://site.com/sitemap.xmlWhere the sitemap lives

Before blocking parameters, check whether any of them are filter pages you deliberately want to rank, for example "men's Nike sneakers". Those pages should get clean URLs and stay open.

robots.txt blocks crawling, not indexing

Disallow stops a crawler from reading a page, but it does not stop Google from showing that URL in search. If other pages link to a blocked URL, it can get indexed without a description - in Search Console this shows up as "Indexed, though blocked by robots.txt".

noindex + crawlable

To remove a page from search, use a meta robots noindex tag or the X-Robots-Tag: noindex HTTP header. And do not block that page in robots.txt: if Googlebot cannot crawl it, it will never see the noindex.

Google has not supported a noindex directive inside robots.txt since 2019. For duplicates that should pass their signals to a main page, the right tool is canonical, not a crawl block.

What to block in robots.txt and what never to block

Block what has no search value and generates lots of URLs: system sections, user accounts, internal search results, sorting and endless filter combinations. Google itself recommends doing this for internal search and unneeded faceted filters.

AI crawlers in robots.txt: block or allow

robots.txt now also decides your site's presence in ChatGPT, Claude, Perplexity and Gemini. Each company runs several crawlers for different jobs, and you can block them selectively.

User-agentOwnerPurpose
GPTBotOpenAICollecting data for model training
OAI-SearchBotOpenAIChatGPT search: showing and citing sites in answers
ClaudeBotAnthropicCollecting data for training Claude models
PerplexityBotPerplexityIndexing for Perplexity search answers
Google-ExtendedGoogleNot a separate crawler but a token: whether content may be used to train and ground Gemini. Has no effect on Google Search

Blocking GPTBot or ClaudeBot keeps your content out of model training, but also lowers the chance that a model knows about your brand. Blocking OAI-SearchBot or PerplexityBot removes you from ChatGPT and Perplexity search answers, which means losing a traffic channel. For most commercial sites that need customers, AI crawlers are worth keeping open.

Keep in mind: robots.txt is an agreement, not protection, and agents triggered by a user (for example, when someone asks ChatGPT to open a page) may not follow these rules, according to the companies' own documentation. Also, AI bots are often blocked not by robots.txt but by hosting or Cloudflare bot protection. How to check your visibility in AI assistants is covered in how to get mentioned in ChatGPT.

I will check your robots.txt and sitemap.xml

Send me your site URL - I will check whether anything important is blocked, whether junk is getting indexed and whether your sitemap is built correctly. The express audit is free.

Message me on Telegram

sitemap.xml rules: what belongs in a sitemap

A sitemap is a list of pages you want to see in search. For Google it is a hint, not an order.

How to set up robots.txt and sitemap.xml: 5 steps

01

List what needs to be blocked

Crawl the site with Screaming Frog or Sitebulb and find out which URL types it generates. For each type, decide: index it, block crawling or add noindex.

02

Write robots.txt

Start with a User-agent: * group, add Disallow lines for system sections and parameters, separate groups for AI crawlers if you decide to block them, and a Sitemap line with the full URL. Save it at the site root.

03

Set up sitemap.xml generation

Turn on the automatic sitemap in your CMS or plugin and make sure it includes only canonical pages returning 200. For a large site, split sitemaps by page type and combine them with an index file.

04

Submit the sitemap in Search Console

In Google Search Console, open Indexing - Sitemaps, paste the file URL and click Submit. The "Success" status means Google has read the file. How to use the other reports is covered in the article on Google Search Console.

05

Check the result

Search Console Settings include a robots.txt report: which file Google fetched, when, and whether there are errors. The URL Inspection tool shows whether crawling of a specific page is allowed. Google caches robots.txt for up to 24 hours, so changes do not apply instantly.

Common robots.txt and sitemap.xml mistakes

Checklist: robots.txt and sitemap.xml

Frequently asked questions

Does every website need a robots.txt file?

No. Without robots.txt, Google crawls the whole site. But online stores and sites with filters almost always need one, so crawlers do not waste time on thousands of technical URLs.

How do I keep a page out of the index: robots.txt or noindex?

Use noindex. robots.txt only blocks crawling, and a blocked page can still be indexed without a description. A page with noindex must stay crawlable in robots.txt, otherwise Google will not see the directive.

Does a small website need a sitemap.xml?

For a site with a few dozen pages and good internal linking, a sitemap is not critical, but it does no harm. For new and large sites, sitemap.xml is strongly recommended.

What are the main robots.txt best practices for WordPress?

WordPress generates a virtual robots.txt automatically. You can change it through an SEO plugin (Yoast, Rank Math) or by uploading your own file to the site root. Typically you block /wp-admin/ with Allow for /wp-admin/admin-ajax.php, internal search and unneeded parameters.

Does blocking AI crawlers in robots.txt hurt SEO?

Blocking GPTBot, ClaudeBot or Google-Extended does not affect Google rankings. But blocking OAI-SearchBot and PerplexityBot removes the site from ChatGPT and Perplexity search answers, so for a commercial site it means losing potential customers.

Summary

robots.txt controls where crawlers go, sitemap.xml controls what they should find, and noindex and canonical control what ends up in the index. Block system sections, search and unneeded filters, but never CSS, JS or the whole site. Keep only canonical 200 pages in your sitemap and check the Search Console reports every month.

I will set up your technical SEO

robots.txt, sitemap, indexing, duplicates and speed - as part of monthly SEO from $450/month. We start with a free express audit.

Send a request

Leave a request

Describe your task - I will reply within a few hours and suggest where to start in your case.

Thank you, your request has been received. I will get back to you shortly.
DZ

Dmytro Zaichenko

Freelance SEO Specialist · 13+ years of experience · 70+ projects · zaichenko.pro