Start your 7-day free trial, card not charged until it ends

Robots.txt monitoring

Robots.txt monitoring that catches a blocked site

Robots.txt monitoring checks the file that tells search engines what they may crawl. TLDTrack reads robots.txt on the sites you switch it on for, lists every path it blocks for each crawler, and alerts you when the file goes missing, blocks the site or picks up a new block.

Start your free trial See how it works

Card not charged until your trial ends

01 · Why it matters

One line can hide a whole site

Nothing on the page looks broken when robots.txt goes wrong. The site just starts dropping out of search.

The staging file goes live

A site launches with the staging robots.txt still in place, User-agent: * followed by Disallow: /, and search engines stop crawling it.

A plugin or migration rewrites it

SEO plugins, hosting moves and CMS updates can all replace the file, and a new block on /blog/ or /shop/ goes unnoticed.

The file disappears or breaks

A deploy drops it, or the server starts returning an error for it. Google stops crawling a site while its robots.txt returns a server error.

02 · Coverage

What the check tells you

There is nothing to type in. The check reads the file for you.

Missing

Is there a robots.txt?

If the file is missing, cannot be read, or the server returns a web page instead of a robots file, you get an alert.

Blocked

Is the site or the homepage blocked?

The rules are read the way Google reads them, from the host your homepage redirects to: the Googlebot group first, then the * group, the longest matching rule wins, and * and $ wildcards count. A whole-site block or a blocked homepage raises an alert.

New blocks

Has anything new been blocked?

When a Disallow rule appears that was not there at the last check, for any crawler, you get an alert that names the path and the crawler.

Every block

Where is the site blocked?

Each check lists every path the file blocks, crawler by crawler, which of the main AI crawlers it shuts out, and the Sitemap lines it points to.

Pair it with the "Homepage blocked from search" check, which also looks for a noindex tag or header on the homepage.

03 · How it works

What happens once it is on

01

TLDTrack reads the file on a schedule

Daily by default, or hourly, every 6 hours, every 12 hours or weekly, for one site or every site you look after.

02

It works out what is blocked

For Googlebot, for all crawlers, and for every named crawler in the file, including AI crawlers such as GPTBot and ClaudeBot.

03

You hear about it once

When something goes wrong, an alert goes out by email and push, and to Slack or Teams on paid plans. You are not alerted again at every check while it stays the same.

04 · Pricing

Simple, transparent pricing

Every paid plan starts with a 7-day free trial. Cancel during the trial and you won't be charged.

Free forever

Free

Monitor one site, free forever

$0/mo

1 website + 4 extra monitors

No credit card needed

  • Uptime checks every 5 minutes
  • Response time tracking
  • Domain expiry alerts
  • SSL certificate expiry alerts
  • Email alerts
Start free

7 days free

Starter

For individuals & small portfolios

$11/mo

Up to 5 websites

250 monitoring credits / month

  • Uptime, SSL, DNS & blacklist
  • Content & visual change monitoring
  • Accessibility (WCAG 2.2) monitoring
  • Email health & renewal alerts
  • PageSpeed & malware scanning
  • Free WordPress plugin
  • WHM & cPanel monitoring
  • Bulk import & export
  • Security, SEO & compliance scans
  • AI integration (Claude & ChatGPT)
Start free trial

Most popular

Pro

For active managers & agencies

$29/mo

Up to 25 websites

1,250 monitoring credits / month

  • Everything in Starter
  • Up to 5 team members
  • Client report builder
  • AI search visibility
  • Priority support
  • Security, SEO & compliance scans
  • AI integration (Claude & ChatGPT)
Start free trial

7 days free

Agency

For large portfolios & teams

$89/mo

Up to 100 websites

5,000 monitoring credits / month

  • Everything in Pro
  • Up to 20 team members
  • Agency CRM & white-label reports
  • Security, SEO & compliance scans
  • Dedicated support
  • AI integration (Claude & ChatGPT)
Start free trial

Bigger than 100 websites? Our Enterprise plan adds unlimited sites, custom data residency, an SLA, dedicated onboarding and custom integrations. Talk to us

*SMS alerts are an optional add-on on paid plans for UK mobile numbers, paid for with prepaid SMS credit packs. Prices exclude any applicable sales tax. Your card won't be charged during a paid-plan trial. Cancel anytime.

05 · Questions

Frequently asked questions

Do I have to write any rules?

No. The check reads robots.txt itself and works out what it blocks. There is nothing to type in.

Will it alert me if robots.txt goes missing?

Yes. A missing file, a file the server cannot return, or a web page served in its place all raise an alert.

Does it look beyond the homepage?

Yes. It reads every rule in the file and lists each path it blocks. It alerts when the whole site or the homepage is blocked, and when a new block appears anywhere, such as /blog/.

Will TLDTrack tell me every time robots.txt changes?

It tells you when a change matters: a new block, a missing or broken file, or a blocked site or homepage. Removing a block or editing comments does not raise an alert, but the current rules are always shown.

Does it show which AI crawlers are blocked?

Yes. Each check names the main AI crawlers, such as GPTBot, ClaudeBot and PerplexityBot, that the file blocks from the whole site.

Does it use credits?

No. The robots.txt check uses no credits.
Mark Grice, founder of TLDTrack

Mark Grice, founder of TLDTrack. Runs FullyCoded, a Cornwall web agency, and built this to keep 500+ client sites in front of him every day.

What happens next

Check robots.txt on every site

Do it yourself

Start your free trial

Add your sites, switch the check on and hear about it the day a site is blocked or the file goes missing. Your card is not charged for 7 days.

Start your free trial

Talk it through

Arrange a call with Mark

If you would rather talk through how this works across every site you look after, we can go through it together.

Book a call

See every check TLDTrack runs