Crawler files

What robots.txt and sitemaps say about a site

Two small public files, robots.txt and sitemap.xml, describe how a site wants search engines to see it. Reading them takes seconds.

Reading robots.txt

Robots.txt lists paths crawlers should avoid and usually points to the sitemap. A coherent file suggests someone maintains the site's search presence.

Reading the sitemap

A sitemap lists the pages the operator considers important, often with modification dates. If a page you were sent is absent, it may be old, hidden or not part of the site's intended content.

Freshness signals

Recent modification dates suggest active maintenance. A sitemap untouched for a long time on a site claiming current Dark Matter information is worth noting.

They are not access controls

Robots.txt is a request to crawlers, not a security barrier. Never treat listed paths as secret or protected.