Reading robots.txt
Robots.txt lists paths crawlers should avoid and usually points to the sitemap. A coherent file suggests someone maintains the site's search presence.
Reading the sitemap
A sitemap lists the pages the operator considers important, often with modification dates. If a page you were sent is absent, it may be old, hidden or not part of the site's intended content.
Freshness signals
Recent modification dates suggest active maintenance. A sitemap untouched for a long time on a site claiming current Dark Matter information is worth noting.
They are not access controls
Robots.txt is a request to crawlers, not a security barrier. Never treat listed paths as secret or protected.