How to create a robots.txt file

A robots.txt file tells search engine crawlers which parts of your website they should and should not crawl. This guide explains what robots.txt does, how to create one, where to place it, and how it differs from other SEO controls.

What is a robots.txt file?

A robots.txt file is a plain-text file that sits at the root of your website. When a search engine crawler visits your site, it looks for this file first to understand which pages or directories it should crawl and which it should avoid.

The file uses a simple set of directives. The most common are User-agent, which identifies which crawler a rule applies to, and Allow or Disallow, which tell the crawler what it may or may not crawl. You can also include a Sitemap directive to help crawlers discover all the URLs on your site.

It is important to understand that robots.txt provides instructions to compliant crawlers. It is not a security mechanism and does not prevent people from accessing a URL directly.

How to create a robots.txt file

1. Enter your website URL.
2. Choose whether to allow all crawlers or block all crawlers.
3. Add any directories you want crawlers to avoid, such as an admin area or checkout pages.
4. Add your sitemap URL so crawlers can discover your site's URLs.
5. Review the generated file and check the validation panel for any issues.
6. Download robots.txt.
7. Upload it to the root directory of your website.

A basic robots.txt file that allows all crawlers and points to a sitemap looks like this:

User-agent: * Allow: /

Sitemap: https://example.com/sitemap.xml

Where should robots.txt be placed?

Your robots.txt file should be placed at the root of your website so it is accessible at a URL like https://example.com/robots.txt. Only one robots.txt file is used per domain, and it must be at the root for crawlers to find it.

For example, if your website is https://example.com, the file must be available at https://example.com/robots.txt. Placing it in a subdirectory like https://example.com/blog/robots.txt will not work.

What should I put in robots.txt?

The most common directives in a robots.txt file are:

• User-agent — Identifies which crawler a rule applies to. An asterisk (*) means all crawlers. You can also target specific crawlers like Googlebot or Bingbot.
• Allow — Tells crawlers they may crawl a specific path. For example, Allow: / permits crawling of the entire site.
• Disallow — Tells crawlers not to crawl a specific path. For example, Disallow: /admin/ asks crawlers not to crawl pages under /admin/.
• Sitemap — Points crawlers to your XML sitemap so they can discover all the URLs on your site.

A typical robots.txt file might allow all crawlers to access the site while blocking a few sensitive directories, and include a sitemap URL at the end.

Does robots.txt block Google?

robots.txt controls crawling, not indexing. If you block a page with robots.txt, Google's crawler will not visit that page. However, Google may still index the page using other signals, such as links from other sites, and it may appear in search results without a description.

If you want to prevent a page from being indexed, use the noindex directive on the page itself, for example with a meta robots tag or an X-Robots-Tag HTTP header. Do not rely on robots.txt to keep a page out of Google's index.

Also remember that robots.txt is not a security mechanism. It tells compliant crawlers what they should not crawl, but it does not prevent people from accessing a URL directly.

Robots.txt vs noindex

robots.txt and noindex serve different purposes:

• robots.txt controls crawling. It tells search engine crawlers which pages they should not visit. It is useful for preventing crawlers from accessing pages that should not be crawled, such as admin areas or search result pages.
• noindex controls indexing. It tells search engines not to include a page in their index, so the page will not appear in search results. It is applied at the page level using a meta robots tag or an X-Robots-Tag HTTP header.

If you want to prevent a page from appearing in search results, use noindex. If you want to prevent crawlers from visiting a page altogether, use robots.txt. In some cases you may use both, but be careful: if you block a page with robots.txt, crawlers will not be able to see the noindex directive on that page, so the page may still be indexed using other signals.

You can use the Robots.txt Generator to create a valid robots.txt file for your website, and the SERP Preview tool to check how your pages could appear in search results.