Easily generate a robots.txt file to control how search engines crawl and index your website.
Crawler access workflow
Robots.txt is a crawl instruction file, not a privacy or security layer. It helps well-behaved crawlers understand which paths they may fetch. Because the file is public, never use it to hide sensitive URLs. Use authentication and server access control for private content.
Most marketing and product sites should allow the main site and block only private, duplicate, or utility paths such as admin areas, internal search result pages, temporary files, and checkout states. Overly broad disallow rules can remove important pages from discovery.
AI crawlers have different purposes. Some fetch pages for search answers, some operate as user-triggered agents, and some collect training data. Decide which categories match your business model before blocking everything. For many publishers, allowing AI search crawlers while restricting training crawlers is a more precise policy.
Include the sitemap URL when possible, test the file after deployment, and keep comments short but useful. Revisit rules quarterly because crawler names change, new AI user agents appear, and old migration paths can become accidental crawl traps.
Not reliably. A URL blocked from crawling can still be discovered from links. Use noindex on crawlable pages when the goal is to keep them out of search results.
Yes. Add a User-agent group for that crawler and set the rule you want for that group.