Free Robots txt Generator with AI Crawler Controls

What this robots txt generator does

A robots.txt file is a plain text file at the root of your website (for example https://www.example.in/robots.txt) that tells crawlers which parts of the site they may visit. Search engines check it before crawling, and so do most AI crawlers. A good file keeps bots away from pages that waste crawl effort, such as carts, admin screens and endless filter combinations, while leaving everything you want found wide open.

This robots txt generator writes the file for you. Pick a starting preset, adjust the paths, choose how each major search and AI crawler should be treated, add your sitemap, and copy or download the result. It follows the Robots Exclusion Protocol standard (RFC 9309), removes duplicate lines, groups blocked bots neatly and warns you about risky choices such as blocking Googlebot or your CSS and JavaScript.

How the rules work

• User-agent names the crawler a group of rules applies to. * means every crawler without its own group.
• Disallow asks crawlers not to fetch matching URLs. An empty Disallow: means nothing is blocked.
• Allow creates an exception inside a blocked folder, such as allowing /wp-admin/admin-ajax.php while blocking /wp-admin/.
• Wildcards: * matches any characters and $ marks the end of a URL, so /*.pdf$ matches every PDF.
• Most specific wins. Google applies the longest matching rule; when an Allow and a Disallow match equally, Allow wins. A crawler that has its own group ignores the * group completely.
• Sitemap lines point crawlers to your XML sitemap and must be full URLs.

How to use the tool

1. Enter your website address so presets can fill in the sitemap URL.
2. Choose a preset: WordPress, WooCommerce, Shopify-style, Allow everything, or Block all for a staging site.
3. Review the Disallow and Allow lists and add or remove paths, one per line.
4. Decide on AI crawlers. Use Block AI training bots for a quick setting, or choose Allow or Block for each crawler individually.
5. Check the sitemap URL actually opens in your browser.
6. Copy the output or download robots.txt, then upload it to the root folder of your domain. On WordPress, many SEO plugins include a robots.txt editor; on Shopify, edit robots.txt.liquid in your theme.
7. Open yoursite.com/robots.txt to confirm it is live, and check the robots.txt report in Google Search Console.

Worked examples

A WooCommerce store selling home decor

Say a store in Jaipur sells handmade decor with price filters and sorting options. Every combination of ?orderby= and ?min_price= creates a new URL with the same products, and crawlers can spend a lot of time on them. The WooCommerce preset blocks cart, checkout, account, internal search and those sorting and price parameters, keeps admin-ajax open for features that depend on it, and points to the sitemap from Yoast or Rank Math.

A D2C brand on Shopify

Shopify generates a sensible robots.txt automatically, so most stores never need to touch it. The Shopify-style preset is useful as a reference, or for brands that want to add rules for AI crawlers through the robots.txt.liquid template. Avoid deleting Shopify’s default rules unless you know why they are there.

A publisher or consultant deciding on AI crawlers

A finance blog or an education business may want its content to appear in AI search answers but not be used to train AI models. A reasonable middle path is to block training-focused crawlers such as GPTBot, Google-Extended, ClaudeBot and CCBot, while leaving search-focused ones such as OAI-SearchBot, Claude-SearchBot and PerplexityBot allowed. Blocking Google-Extended does not change how you appear in Google Search. This is a business decision rather than a technical one; the tool simply makes either choice easy.

A staging site for a redesign

When a developer builds a new version of a site on a staging subdomain, the “Block all” preset stops well-behaved crawlers from wandering in. Combine it with password protection, and remember to replace the file before launch. A forgotten Disallow: / on a live site is one of the most common reasons a relaunched site disappears from search.

An exporter with a slow shared server

If Bing’s crawler is putting load on a small server, a modest Crawl-delay such as 5 or 10 seconds can help. Google ignores Crawl-delay and sets its crawl rate based on how quickly and reliably your server responds.

Tips and common mistakes

• Robots.txt is not a privacy tool. The file is public and only a request. Never list secret folders in it; protect them with passwords instead.
• Blocking is not the same as removing from Google. A disallowed page can still be indexed if other sites link to it. To keep a page out of search results, allow crawling and add a noindex meta tag.
• Don’t block CSS, JavaScript or images that your pages need. Google renders pages like a browser, and blocked resources can make a page look broken to it.
• One file per host. www.example.in, example.in and shop.example.in each read their own robots.txt.
• Paths are case-sensitive. /Cart/ and /cart/ are different rules.
• Changes are not instant. Google generally caches robots.txt for up to about a day, so allow some time before checking the effect.

FAQs

Where do I put the robots.txt file?

At the root of your domain, so it opens at https://www.yoursite.com/robots.txt. A file in a subfolder is ignored. On most hosting you upload it to the same folder as your homepage files.

Does blocking GPTBot or ClaudeBot hurt my Google rankings?

No. Those crawlers belong to OpenAI and Anthropic and have nothing to do with Google Search. Blocking Google-Extended also does not affect Google Search; it only controls use of your content for Google’s Gemini models.

Should I block AI crawlers?

It depends on your goals. If you want your brand and content to be cited in AI answers, keep search-focused AI crawlers allowed. If you are concerned about your content being used for model training, block the training crawlers. Many businesses choose a mix.

What happens if I have no robots.txt file?

Crawlers treat a missing file as permission to crawl everything. That is fine for small sites, but a file with your sitemap and a few sensible exclusions is still good practice.

Is Crawl-delay worth using?

Only if a crawler that respects it, such as Bingbot, is overloading your server. Googlebot ignores it, so it will not slow down Google.

A correct robots.txt is one small part of technical SEO; crawl budget, indexing, site speed and structured data all play a role too. If you want a full technical check of your website, explore GrowthLife’s SEO services, or try our other free marketing tools, including a schema markup generator and a meta description checker.