Skip to content
🤖

Robots.txt Analyzer

SEO

Analyze robots.txt files to check crawler directives, blocked paths, and sitemap references.

0
0
Edit robots.txt content
Preview / Rules
ALLOW*/public/
DISALLOW*/admin/
DISALLOW*/private/
ALLOW*/api/public
DISALLOWGooglebot/private/
ALLOWGooglebot/

Editing robots.txt visually and watching a live crawler simulation update is one workflow, meant for building or actively changing rules, but auditing a file that's already live and presumably working is a different task, checking what's actually being blocked, confirming a sitemap reference is present, without necessarily changing anything at all. This tool analyzes an existing robots.txt file to check crawler directives, blocked paths, and sitemap references, built for auditing a file that's already deployed rather than editing or building one from scratch.

See more about the Robots.txt Analyzer

Useful for auditing a live site's robots.txt to confirm exactly which paths are actually being blocked, checking that a sitemap reference is present and correctly formatted, or reviewing an inherited robots.txt file's directives before deciding whether anything actually needs changing.

Key features

  • Clean interface
  • Fast processing
  • No signup required
  • Works offline

Related SEO tools

Broken Link Checker

SEO

Slug Health Checker

SEO

Google SERP Preview

SEO

Word Density Analyzer

SEO

Duplicate URL Detector

SEO

Open Graph Preview

SEO

Related Blog Posts

Quick answers for Robots.txt Analyzer

Does it flag directives it doesn't recognize?
Yes, running Validate checks every line against the known directive set (User-agent, Allow, Disallow, Sitemap, Crawl-delay, Clean-param) and reports "Unknown directive" by line number for anything else.
How does it check Sitemap lines?
It confirms the value after Sitemap: starts with http, flagging it as an error if you paste a relative path instead of a full URL.
What does the rule breakdown look like?
Below the editor, every Allow and Disallow line is listed with its user agent and path, color coded green for allowed and red for disallowed, so crawler directives and blocked paths are easy to scan.