What is a Robots.txt File?
A `robots.txt` file sits in the root directory of a web server (e.g. `example.com/robots.txt`). It uses the Robots Exclusion Protocol to inform search engine bots (like Googlebot) which areas of a website should not be crawled.
Generate custom crawler rules using our Robots.txt Generator Tool.
Key Robots.txt Directives
- `User-agent: *`: Applies rules to all web crawlers.
- `Disallow: /admin/`: Prevents bots from crawling admin panel paths.
- `Allow: /`: Explicitly permits crawling public pages.
- `Sitemap:`: Specifies the absolute URL location of your XML sitemap.
Generating a Robots.txt File
- Open the Robots.txt Generator.
- Specify user-agents and disallowed paths.
- Click "Generate Robots.txt" and save the output to your domain root folder.