Free online robots.txt generator. Configure User-agent and Allow/Disallow rules to generate a standard robots.txt file.
TL;DR: Free online robots.txt generator. Configure User-agent and Allow/Disallow rules to generate a standards-compliant robots.txt, with sitemap declaration support. Search crawler control made easy. Local processing.
For local tools, open DevTools → Network and search your input — it should not appear in requests.
Usage Guide
robots.txt lives at your site root and tells crawlers what to fetch and what to skip — the first config file of SEO. This tool generates it visually as groups (User-agent) + rules (Allow/Disallow) + a Sitemap line, with live preview.
1. Generate a robots.txt
- Add a group: the wildcard * governs all crawlers; create separate groups for specific bots (GPTBot, Googlebot, ...).
- Add Allow/Disallow rules per group: paths match by prefix, e.g. /admin/.
- Fill in the sitemap URL (helps engines find your URL list quickly), then copy the result into robots.txt at your site root.
2. Two mental models to get right
- robots.txt is a request, not a wall: compliant engines obey, malicious crawlers ignore it. Real privacy requires authentication — robots only reduces pointless crawling.
- robots.txt is public: every Disallow announces "something is here" — listing admin paths hands attackers a map.
- "Blocked from crawling" ≠ "blocked from indexing": a Disallowed URL with external links can still appear in results (without a snippet). To truly prevent indexing, use noindex.
3. The two costliest accidents
Shipping a staging Disallow: / to production makes the whole site vanish from search for weeks. And assuming Disallow prevents indexing, only to find pages listed without snippets. A robots.txt check in the release checklist is far cheaper than either.
FAQ
Neither — there is no sign-up system. Rule configuration and file generation all happen in your own browser.
Allow/Disallow rules grouped by User-agent, Crawl-delay directives, and a Sitemap declaration with an absolute URL. A live preview shows the file as you edit; the output follows standard syntax and takes effect once placed in your site root.
No. robots.txt is a gentlemen's agreement that malicious crawlers can ignore. Protect sensitive content with authentication, unlinked URLs, and proper cache behavior — never rely on robots.txt to hide it.
There is no single answer; it depends on each crawler's implementation, and rule order and specificity affect the outcome. After editing, check real crawl behavior via Search Console or a public testing tool.
Confirm robots.txt at your domain root is publicly reachable, the Content-Type is sensible, and the file has no stray BOM. Staging and production rules can differ — the production root path is what counts.
Keep the file in version control and on your release checklist, back up the old copy before major changes, and make sure a site-wide ban won't block landing pages by accident.
Related Tools
Related Articles
Allow or Block AI Crawlers: GPTBot, ClaudeBot, PerplexityBot — a robots.txt Decision Guide
Training crawlers and search crawlers are different agents — blocking GPTBot does not remove you from ChatGPT Search. A crawler-by-crawler table aligned with this site's robots.txt generator AI preset, a three-type decision framework, copy-paste rules, and how to measure payoff after allowing.
llms.txt, Explained: A Site Manual for LLMs — and Whether Yours Deserves One
llms.txt is a 2024-proposed markdown file at your site root that tells large models what's worth reading. This post breaks down the spec structure (H1 + summary + tiered links), how it divides labor with robots.txt and sitemap, multi-file digests (zh / full), common failure modes, and a real auto-generated implementation — with a skeleton and annotation recipes you can copy.
TOTP Two-Factor Authentication Guide: otpauth URI, Algorithms, and Local Debugging
How RFC 6238 TOTP works: algorithms, digits, period, otpauth URIs, countdown windows, and debugging 2FA with a browser-local generator.