Key facts
| Tool type | Free robots.txt tester for AI crawlers |
| Crawlers checked | GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and others |
| Checks | Per-user-agent allow/block, syntax errors, sitemap references |
| Companion files | llms.txt generator and validator |
| Interpretation | Robots rules are advisory; compliance varies by crawler |
| Content factors | Clear headings, answer-first paragraphs, structured data |
| Free plan | 2 free AI models (plugsky-micro, plugsky-lite), no card required |
| Product status | Live |
TL;DR
- AI crawlers obey their own user-agent rules; a blanket block can hide you from AI answers.
- Test before and after every robots.txt change rather than reading directives by eye.
- GEO is more than crawler access: answer-first content and structure decide citation quality.
- llms.txt helps assistants find your key pages but does not replace robots.txt or sitemaps.
- Re-test after CDN, firewall or bot-management changes, which can override robots rules.
How it works, step by step
- Open the robots.txt tester for AI crawlers and enter your domain.
- Review the per-crawler results: which agents are allowed and which are blocked.
- Check for syntax errors and conflicting rules, especially wildcard blocks.
- Fix unintended blocks in robots.txt and confirm the sitemap line is present.
- Re-test and verify the same crawlers now resolve to allow.
- Publish or update llms.txt for a curated map of your important pages.
- Review content structure — one idea per section, answer first — and re-check monthly.
Try it yourself
Open the robots.txt tester for AI crawlers →
Why AI crawlers need their own check
AI assistants discover content through crawlers that identify themselves with distinct user-agent strings: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, PerplexityBot, Google-Extended and others. A robots.txt written years ago to block aggressive scrapers may block these agents too, which removes your pages from AI answers without any visible error. The tester makes that invisible state visible by evaluating your rules for each crawler and showing exactly which directive decides the outcome.
Reading allow and block results
Start with intent. If you want citations from AI assistants, the search and browsing agents must be allowed. If you want to withhold content from model training while remaining citable, target the training crawlers specifically instead of blocking everything. Watch for rules that are technically valid but operationally fragile: a wildcard disallow for / with a later allow for /articles can behave differently across crawlers. Remember that robots.txt is an advisory convention — reputable crawlers respect it, but your access logs and CDN bot rules are the only real enforcement layer.
Beyond robots.txt: the rest of GEO
Crawler access is necessary, not sufficient. Generative engines extract passages, so pages that answer a question in the first paragraph and structure content with clear headings are easier to quote accurately. Structured data helps machines extract facts about your product, organisation and articles. An llms.txt file gives assistants a curated map of your most important pages, complementing rather than replacing XML sitemaps. Finally, verify your assumptions: check the docs for how Plugsky exposes machine-readable catalogue data, and review the security page if bot management sits between crawlers and your origin. GEO is maintenance, not a one-off task.
Honest comparison
| Signal | What it does | Effort | Does it guarantee citations? |
|---|---|---|---|
| robots.txt allow rules | Permits AI crawlers to fetch pages | Minutes | No, it only removes a blocker |
| XML sitemap | Lists canonical URLs for discovery | Low, automate it | No |
| llms.txt | Curated map for assistants | Low, generate and review | No, convention still emerging |
| Structured data | Makes facts extractable | Medium per template | No, improves machine readability |
| Answer-first content | Makes passages quotable | Ongoing editorial work | No, but it is the biggest lever |
Frequently asked questions
What does GEO stand for?
Generative engine optimization: making content discoverable, readable and quotable for AI assistants and answer engines, as opposed to classic search ranking alone.
Is the robots.txt tester free?
Yes. It is a free browser tool with no account required.
Will allowing AI crawlers hurt my SEO?
No. Allowing legitimate AI crawlers does not replace search indexing; it adds another discovery path. Block only what your policy requires.
Should I block AI crawlers from training?
That is a policy decision. You can allow search and browsing agents while targeting training-only agents, but verify each crawler's purpose in its documentation.
Is llms.txt a ranking factor?
No. It is an emerging convention, not a ranked file. It helps assistants find key pages but does not replace sitemaps, robots.txt or content quality.
Why does robots.txt sometimes get ignored?
It is advisory. Some crawlers or scrapers ignore it, and bot-management or firewall rules at the CDN layer can override behaviour regardless of the file.
How often should I test?
After every robots.txt or CDN change, and on a monthly schedule to catch provider-side crawler updates.
Does Plugsky publish an llms.txt?
Yes. Plugsky publishes llms.txt and llms-full.txt so assistants can discover its catalogue and documentation, and the generator tool builds one for your site.