Updated
Robots.txt is a text file in the root of your website (example.com/robots.txt). It defines which areas crawlers from search engines and other services may visit and which they may not.
Structure
- User-agent: which crawler the rule applies to, * means all
- Disallow: paths that should not be crawled
- Allow: exceptions inside blocked areas
- Sitemap: the address of your XML sitemap
Example: User-agent: * / Disallow: /wp-admin/ / Sitemap: https://example.com/sitemap_index.xml (each directive on its own line).
Robots.txt is not noindex
A blocked page can still end up in the index if other pages link to it. To keep a page out of Google, use noindex and allow crawling. See indexing for more.
AI crawlers
Robots.txt also lets you control AI crawlers such as GPTBot (OpenAI), ClaudeBot (Anthropic) or Google-Extended. If you want to appear in AI answers, do not block them across the board. See GEO.
Digitalman checks robots.txt and crawling on your website. Learn more on our on-page SEO page.
Digitalman can help
Want a technically sound website for Google?
We check indexing, structure, speed and content and implement the fixes for you. The first 60 minutes of consultation are free.