Robots.txt Generator

Introduction

Robots.txt is the first file most search engine crawlers request from your domain root. A single misplaced Disallow rule can block your entire blog, API documentation, or product catalog from crawling — sometimes for weeks before anyone notices in Search Console coverage reports. Generating syntactically correct robots.txt should not require memorizing directive syntax or risking typos in production configs.

Certoflow's Robots.txt Generator builds crawler rules with User-agent declarations, Allow and Disallow paths, optional crawl-delay, and Sitemap URL references. Output is copy-ready plain text for placement at https://yourdomain.com/robots.txt. Processing runs entirely in your browser — internal path structures, staging domains, and client site architectures never upload to a server. Light and dark theme support keeps the form comfortable during site migration work.

Pair with XML Sitemap Generator to create the sitemap file referenced in your robots.txt Sitemap directive.

What this tool does

The generator assembles robots.txt from configurable options:

Feature	Behavior
User-agent	Default * wildcard for all crawlers
Allow all	Permissive mode when no Disallow paths are specified
Disallow paths	One or more path prefixes to block from crawling
Crawl-delay	Optional seconds between requests (non-standard but supported by some bots)
Sitemap URL	Absolute URL to your sitemap.xml file
Output	Plain text robots.txt ready to copy and deploy

Disallow paths are normalized to start with /. Empty path entries are filtered from output.

How it works

Certoflow's generateRobotsTxt function constructs the file line by line:

User-agent: *
Allow: /                    (when allow-all with no disallow paths)
Disallow: /admin            (for each configured path)
Crawl-delay: 10             (optional, when set)
Sitemap: https://example.com/sitemap.xml

When Disallow paths are present, the Allow directive is omitted unless explicitly configured for permissive defaults. The Sitemap directive accepts absolute URLs — required for crawlers to discover your sitemap location.

All generation runs synchronously in JavaScript with no server round trip. Copy output, save as robots.txt, upload to your site root via FTP, SSH, or deployment pipeline.

Example output for a blog with admin area blocked:

User-agent: *
Disallow: /wp-admin/
Disallow: /private/

Sitemap: https://example.com/sitemap.xml

Real-world examples

WordPress admin protection

Standard WordPress deployment blocks https://example.com/wp-admin/ from crawlers while allowing public content. Add Disallow for /wp-admin/ and /wp-includes/ on that domain, reference sitemap generated by XML Sitemap Generator.

Staging environment lockdown

Staging site at staging.example.com should not be crawled. Set Disallow: / for all paths. Production robots.txt uses Allow: / with specific exclusions only.

API documentation site

Public docs at https://example.com/docs/ but internal API explorer at https://example.com/docs/internal/. Disallow /docs/internal/ while allowing the rest. Submit sitemap listing public doc URLs only.

E-commerce filter pages

Faceted navigation generates infinite parameter combinations. Disallow https://example.com/catalog? or specific filter path patterns. Combine with Canonical Tag Generator on remaining indexable pages.

Crawl budget management

Large site with low-value tag archive pages. Disallow https://example.com/tag/ paths to direct crawler attention toward product and content URLs listed in sitemap.

Common mistakes

Treating robots.txt as an indexing blocker. Disallow prevents crawling, not necessarily indexing. URLs linked externally may still appear in search results. Use <meta name="robots" content="noindex"> for stronger index control.

Blocking CSS and JavaScript. Google needs render resources for proper indexing. Do not Disallow https://example.com/assets/ or https://example.com/static/ unless you understand the rendering impact.

Forgetting the Sitemap directive. Crawlers discover sitemaps through robots.txt, Search Console submission, and internal linking. Include the Sitemap line for reliable discovery.

Using relative Sitemap URLs. The Sitemap directive requires an absolute URL including protocol: https://example.com/sitemap.xml.

Typos in Disallow paths. Disallow: /admin blocks /admin and everything beneath it. Verify path prefixes match your URL structure exactly — trailing slashes matter for some servers.

Relying on crawl-delay for Google. Google ignores crawl-delay. Bing and Yandex may respect it. Do not depend on crawl-delay for Google crawl rate management — use Search Console settings instead.

Related tools

Within the SEO category:

• XML Sitemap Generator — create the sitemap referenced in robots.txt
• Canonical Tag Generator — page-level duplicate content signals
• Meta Tags Generator — noindex meta tags for pages robots.txt cannot protect

Cross-category links:

• Slug Generator — clean URL paths before defining Disallow rules
• URL Encoder — encode special characters in sitemap URLs
• Meta Tags Generator — robots meta directives complementing robots.txt

Browse all SEO Tools at /seo.

FAQ

Is my robots.txt configuration sent to a server?

No. Generation runs locally in JavaScript. Path rules and domain URLs never leave your device.

Where does robots.txt go?

At your site root: https://example.com/robots.txt. It must be accessible via HTTP GET.

Does robots.txt block Google from indexing pages?

Disallow prevents crawling. Pages may still be indexed if linked from other sites. Use noindex meta tags to prevent indexing.

Can I target specific crawlers?

This generator uses User-agent: * for all bots. Custom user-agent blocks may be added manually to copied output for bot-specific rules.

Does Certoflow ping search engines?

No. Deploy robots.txt to your server. Google discovers changes on subsequent crawls.

Can I use this offline?

Yes, after the page loads.

Does Certoflow support dark mode?

Yes. Toggle light and dark themes for comfortable editing.

What is crawl-delay?

Optional seconds between crawler requests. Not supported by Googlebot. Some other bots may honor it.

Should I disallow my sitemap URL?

No. The sitemap should remain crawlable. Reference it with the Sitemap directive, do not Disallow it.

How do I test robots.txt?

Use Google Search Console robots.txt Tester after deployment, or fetch https://yourdomain.com/robots.txt directly in a browser.