Interactive controls could not load. Reload this page to try again; the content and guides below are still available.
Reload pageWeb Collection Studio
Check public or pasted robots.txt rules for free against up to 100 same-origin URLs. Review matched lines, user-agent decisions and sitemap declarations.
YOUR WORKSPACE
Check public or pasted robots.txt rules for free against up to 100 same-origin URLs. Review matched lines, user-agent decisions and sitemap declarations.
The sample blocks /private/report for general crawlers and allows /private/public. ResearchBot uses its specific group.
Rule decisions do not grant permission or prove page availability. HTTP 4xx means no usable rules; network errors or 5xx pause online collection. Crawl-delay is a nonstandard extension; its declared value is displayed.
No. Only robots.txt is fetched or pasted rules are analyzed.
Use a separate check for each origin, with up to 100 URLs per check.
Use this KitNelo tool for free without an account. Collection and source parsing use the server; access rules and hosting limits apply.
Up to 100 URLs on one origin. Does not visit test pages or grant permission to collect them.
Read KitNelo data and privacy detailsFetch public robots.txt or paste its contents to inspect allow, disallow and sitemap declarations. Review the matched rule lines alongside the results; this evaluates the supplied rules and does not visit the individual test pages.
Enter a user agent and up to one hundred URLs on the same origin to compare rule decisions. Check that the tested origin and agent match your intended crawl, since a rule result does not grant collection permission.
Choose Googlebot as the user agent and supply the paths you want to evaluate. Inspect the matching group and rule line for each decision; the result describes robots directives rather than guaranteeing crawling, indexing or access to a page.
