robots.txt Generator

Create a robots.txt file to control how search engines crawl your website. Configure user-agents, allow/disallow rules, and sitemap URL.

Preset Templates

Configuration

User-agent

Directives (Allow/Disallow)

Sitemap URL

Crawl-delay (seconds, optional)

Note: Most major crawlers (Google, Bing) ignore Crawl-delay. Used mainly for smaller crawlers.

Preview

User-agent: *
Disallow: /admin/

Sitemap: https://example.com/sitemap.xml

What is robots.txt?

The robots.txt file tells search engine crawlers which pages or directories they can or cannot request from your site. It is placed in the root of your website (e.g., https://yoursite.com/robots.txt) and follows a simple directive-based format.

User-agent and Directives

User-agent identifies the crawler (e.g., * for all, Googlebot for Google). Allow and Disallow specify paths that crawlers may or may not access. Disallow rules help you block admin panels, API endpoints, and other areas you do not want indexed.

Sitemap and Crawl-delay

Including your Sitemap URL helps search engines discover your pages efficiently. Crawl-delay (in seconds) limits request frequency, but most major search engines ignore it; it is mainly useful for smaller bots.

Related Resources

How to Use the robots.txt Generator

Select a preset template to start quickly — "Allow All" for open crawling, "Block All" for staging sites, or "Block Common Directories" to exclude admin and API paths.
Configure the User-agent field to target specific crawlers (e.g., * for all bots, Googlebot for Google, Bingbot for Bing).
Add Allow and Disallow rules for specific paths. Disallow sensitive directories like /admin, /api, /private, and staging environments.
Enter your Sitemap URL (e.g., https://yoursite.com/sitemap.xml) to help search engines discover all your pages efficiently.
Copy the generated robots.txt content or download it as a file. Place it at the root of your website (e.g., https://yoursite.com/robots.txt).

Common Use Cases

Blocking search engines from indexing admin panels, login pages, and internal dashboards that should not appear in search results.
Preventing crawlers from accessing API endpoints and webhook URLs that waste crawl budget and expose internal routes.
Setting up staging and development environments with a "Disallow All" rule to prevent accidental indexing.
Directing crawlers to your XML sitemap for faster and more complete indexing of important pages.
Managing crawl budget for large sites by blocking low-value pages like search results, filters, and pagination.
Configuring bot-specific rules to allow Googlebot while blocking aggressive scrapers or AI training crawlers.

Frequently Asked Questions

Does robots.txt prevent pages from being indexed?

Not exactly. Robots.txt tells crawlers not to crawl specific paths, but if other pages link to a disallowed URL, search engines may still index it (showing the URL without a snippet). To fully prevent indexing, use a noindex meta tag or X-Robots-Tag header in addition to robots.txt rules. Learn more with our Meta Tag Generator.

Where should I place the robots.txt file?

The file must be at the root of your domain: https://yoursite.com/robots.txt. Subdomains need their own robots.txt files. In Next.js projects, you can generate it programmatically using a robots.ts file in the app directory.

Should I include a crawl-delay directive?

Most major search engines (Google, Bing) ignore the crawl-delay directive and manage their own crawl rates. It can be useful for smaller bots that respect it. For enterprise SEO strategy and technical optimization, explore our SEO services or check your site's health with our other SEO tools.