🤖 robots.txt Generator

Control crawlers. Block AI. Protect your content.

⚡ Quick Presets

📍 Sitemap URL

🔍 Search Engines

🔵

Googlebot

Google Search crawler

🟢

Bingbot

Microsoft Bing crawler

🟣

DuckDuckBot

DuckDuckGo crawler

🤖 AI Crawlers

Block AI companies from scraping your content for training. Toggle on = blocked.

💚

GPTBot

OpenAI's crawler

💚

ChatGPT-User

ChatGPT browsing feature

🧡

anthropic-ai

Anthropic's crawler

🔷

Google-Extended

Google AI training

🟦

CCBot

Common Crawl (AI training)

🟠

Bytespider

ByteDance/TikTok AI

🔴

ClaudeBot

Anthropic Claude crawler

🟣

PerplexityBot

Perplexity AI crawler

🟤

cohere-ai

Cohere crawler

🔵

FacebookBot

Meta AI training

📝 Custom Rules

No custom rules yet

⏱️ Crawl Delay (optional)

seconds

📄 robots.txt

C Copy S Share T Theme R Reset

aikemist.com · Made by Alkemist

Copied to clipboard!