What does the result Allowed and Disallowed mean?

Allowed means that the robots.txt rules do not block this path for the chosen bot. Disallowed means it is blocked.

Does the tool take domain redirects into account?

Yes. The tool detects the final domain and fetches robots.txt from the target host.

How does matching of Disallow and Allow rules work?

The tool looks for the best match. The longest matching rule wins. Allow wins a tie.

Why do I see syntax errors in validation?

Errors appear if directives are in an incorrect format or placed before User agent. Highlighting makes corrections easier.

What is the purpose of the sitemap status table at the end of the results?

Robots.txt often contains Sitemap directives. The table checks if the server returns the correct HTTP status for these addresses.

Robots.txt Tester

Check if your robots.txt file has correct syntax and whether a specific path is allowed for a selected bot.

What does Robots.txt Tester do?

Robots.txt Tester fetches the robots.txt file from your domain and analyzes its rules. The tool detects redirects and tests the rules on the final hosting. This helps you avoid mistakes when your domain redirects to a language version.

You can select a popular User agent, such as Googlebot. Then the tool checks if the given URL is Allowed or Disallowed. In the results, you'll see the matching rule and the line number.

The tool also performs syntax validation of the robots.txt file. It shows detected issues and highlights incorrect lines. Additionally, it displays the full content of the robots.txt file in a text field.

How does Robots.txt Tester help SEO specialists and website owners?

A single incorrect rule may block the crawl and indexing of key site pages. This often means a drop in visibility and loss in organic traffic. By testing a single path, you can quickly spot risks before deploying changes.

In practice, you save time during the audit. Instead of manually analyzing rules, you get a clear decision and match. This cuts verification time from minutes to seconds and reduces the number of mistakes.

Check whether an address is accessible to the search engine bot.
Preview the matching rule and line number in robots.txt.
Assessment of syntax validity and fast identification of issues.
Safe reading of robots.txt after redirects to the target domain.
Check HTTP status for sitemap addresses declared in robots.txt.

Typical use cases of Robots.txt Tester

Check if new pages are not blocked by Disallow.
Verify rules after domain migration or CDN implementation.
Test access for different bots, including Googlebot and Bingbot.
Diagnose issues with crawl budget and unexpected exclusion of subpages.
Check if robots.txt does not return HTML instead of a text file.
Quick validation of Sitemap directives and their HTTP statuses.

Comparison of Robots.txt Tester with other tools

Functionality	DiagnoSEO	Other tools
Automatic fetching of robots.txt from the domain	✅	✅
Testing rules for selected User agent	✅	✅
Allowed or Disallowed decision for a specific path	✅	✅
Show matching rule and line number	✅	❌
Highlight syntax errors in robots.txt content	✅	❌
Support for redirects and fetching robots.txt from final domain	✅	❌
Checking HTTP status of sitemap addresses from robots.txt	✅	❌
Display full robots.txt content in a text field	✅	✅

Tips and best practices

Test URLs of category, filter, and pagination pages before publishing changes.
Use precise Allow rules when blocking with broad Disallow patterns.
Keep robots.txt consistent with your website's indexing strategy.
After deployment, check the HTTP status of your robots.txt file and declared sitemaps.
Avoid accidentally blocking resources needed for page rendering.

Most common mistakes

Missing robots.txt file.
Blocking important sections with too broad Disallow.
Incorrect directive syntax and missing colon.
Rules before User agent, making group interpretation difficult.
Unintentionally redirecting robots.txt to HTML or the homepage.
Outdated Sitemap entries or incorrect server response codes.

How to use Robots.txt Tester

Paste the URL of the page you want to test.
Select a User agent from the list, for example Googlebot.
Start the test and check the decision: Allowed or Disallowed.
Review the matching rule and line number in robots.txt.
Check the syntax validation messages and highlighted errors.
At the bottom, check HTTP status for sitemap XML addresses from robots.txt.

Case study

An online store noticed a drop in the number of indexed pages. After testing, it turned out that a Disallow rule was blocking a new category path. The tool showed the match and the line number in robots.txt. After correcting the rule and retesting, the URLs were crawled properly again.

Additionally, the sitemap status table showed a problem with one sitemap file. The server returned an error, so bots could not fetch it. After fixing the sitemap file, monitoring became easier.

FAQ

What does the result Allowed and Disallowed mean?

Allowed means that the robots.txt rules do not block this path for the chosen bot. Disallowed means it is blocked.
Does the tool take domain redirects into account?

Yes. The tool detects the final domain and fetches robots.txt from the target host.
How does matching of Disallow and Allow rules work?

The tool looks for the best match. The longest matching rule wins. Allow wins a tie.
Why do I see syntax errors in validation?

Errors appear if directives are in an incorrect format or placed before User agent. Highlighting makes corrections easier.
What is the purpose of the sitemap status table at the end of the results?

Robots.txt often contains Sitemap directives. The table checks if the server returns the correct HTTP status for these addresses.