Robots.txt Tester

Test URLs against your robots.txt file. Check which pages are allowed or blocked for specific user agents like Googlebot, Bingbot, etc.

Or type custom agent string

Parsed Rules

Enter robots.txt to see parsed rules

How robots.txt testing works

  • User-agent matching: Finds the most specific matching group (exact match > prefix match > *)
  • Longest match wins: Among matching Allow/Disallow rules, the longest path pattern takes precedence
  • Allow overrides Disallow: When paths are equal length, Allow takes precedence
  • Default allow: If no rule matches, the URL is allowed
  • Path matching: Supports wildcards (*) and end-of-string ($) per robots.txt spec

Common user agents

* (all bots)GooglebotGooglebot-ImageGooglebot-VideoGooglebot-NewsBingbotSlurp (Yahoo)DuckDuckBotBaiduspiderYandexBotfacebookexternalhitTwitterbotLinkedInBotTelegramBotWhatsAppSlackbotDiscordbotApplebotia_archiver (Alexa)Mediapartners-Google (AdSense)

Pattern examples

PatternMatches
/All paths
/admin//admin/, /admin/page, /admin/sub/page
/private*/private, /private/, /private-data
*.pdf$/file.pdf, /docs/report.pdf (not /file.pdf?query)
/search?q=*/search?q=test, /search?q=foo&bar=baz

Related tools

More from SEO & Marketing

Browse all 100 tools →

Meta Tag Generator

SEO titles, OG & Twitter tags

SERP & Social Preview

Google & social link previews

UTM Builder

Campaign tracking URLs

Slug Generator

SEO-friendly URL slugs

Build custom software with us

From internal tools to full platforms — we deliver end to end.

Start a Project
Was this tool useful?