Corvo

Web & SEO

Robots.txt Generator

Build a valid robots.txt file — allow or block paths, block specific AI crawlers, add your sitemap. Free, instant, entirely in your browser.

Runs entirely in your browser. Nothing is uploaded.

Crawling
Block AI crawlers optional
Sitemap optional
Ready

What this tool does

Builds a valid robots.txt file — the plain-text file at the root of a site that tells well-behaved crawlers which paths they're welcome to fetch. Block specific folders, allow exceptions within a blocked folder, opt out of specific AI training crawlers by name, and point to your sitemap, all from one form.

How to use it

  1. Choose Custom rules for normal use, or Block everything for a staging site you don't want crawled at all.
  2. List any paths to disallow, one per line — an admin area, a cart, anything not meant for search engines.
  3. Check off any AI crawlers you specifically want to keep out.
  4. Add your sitemap URL if you have one.
  5. Copy or download the result and upload it to your site's root as robots.txt.

A quick reference

User-agent: * applies a rule to every crawler that doesn't have a more specific block of its own. Disallow: /path/ asks crawlers not to fetch anything under that path; a bare Disallow: with nothing after it means nothing is restricted. Allow: carves out an exception within a disallowed path. Sitemap: points crawlers to your sitemap so they can discover pages more efficiently — it can go anywhere in the file and isn't tied to a specific user-agent block.

Questions

Is this Robots.txt Generator free?

Yes, completely, with no signup and no limit on how many files you generate.

Is my input sent anywhere?

No. The file is built entirely in your browser as you type — nothing is sent to a server.

Where does this file actually go?

Upload it to the root of your domain, so it's reachable at yoursite.com/robots.txt exactly — not in a subfolder. That specific location is where crawlers look for it automatically.

Does robots.txt actually stop pages from appearing in search results?

Not reliably by itself — it asks well-behaved crawlers not to crawl a path, but a page can still get indexed (typically with no preview) if something else links to it. To reliably keep a specific page out of search results, use a noindex meta tag on that page instead. Robots.txt is best for managing crawl behavior and load, not for hiding content.

Why would I want to block AI crawlers specifically?

Some site owners want their content indexed by search engines (so people can find the site) but not used to train AI models, and several AI companies now publish a distinct crawler specifically for that purpose (separate from the one used for their AI-powered search or browsing features). Blocking those bots by name lets you draw that line, though it depends on the crawler honestly following the file — robots.txt is a request, not an enforcement mechanism.

Do all crawlers respect robots.txt?

No — it's a voluntary convention, not a technical barrier. Major search engines and most reputable crawlers follow it, but nothing stops a badly-behaved bot from ignoring it entirely. For content you genuinely need to keep private, use real access controls, not robots.txt.

Related tools