Robots.txt is a file that tells search engine crawlers which pages they can or cannot access on your website.

Where should I place robots.txt?

Place robots.txt in your website root directory, accessible at yourdomain.com/robots.txt.

Robots.txt Generator - Create Robots File

FreePrivateInstant

Basic Settings

Allow all crawlers

Allow search engines to crawl your entire site

Sitemap URL

Crawl Delay (seconds)

Time between crawler requests. Leave empty for no delay.

Common Disallow Rules

Block /admin/

Prevent crawling admin pages

Block /api/

Prevent crawling API endpoints

Block private paths

Block /private/, /_next/, and JSON files

Custom Disallow Rules

No custom rules added. Click "Add Rule" to add one.

Generated robots.txt

User-agent: *
Allow: /
Disallow: /admin/
Disallow: /admin
Disallow: /api/
Disallow: /api
Disallow: /private/
Disallow: /private
Disallow: /_next/
Disallow: /*.json$

How to Use

Configure your robots.txt settings above
Download or copy the generated file
Place the robots.txt file in your website's root directory
Verify it's accessible at: yourdomain.com/robots.txt
Submit your sitemap to Google Search Console

Create robots.txt files for SEO

What is Robots.txt Generator?

Generate robots.txt files for your website. Control search engine crawling and indexing behavior. Our free online tool makes it easy to create robots.txt files for seo without installing any software. Simply use your browser to get started instantly.