Robots.txt, noindex and pages without links: how to find out where the difference lies.
Snaglist reads the sitemap and then follows every internal link. If it finds fewer pages than the sitemap lists, the difference is usually due to one of these reasons.
Snaglist always follows robots.txt. If a directory is blocked there, we skip it. The export includes the URLs we didn’t fetch for this reason.
A page that only appears in the sitemap, without a link from another page, is an orphan. Search engines struggle to find it. Snaglist flags these pages separately.
Old URLs that redirect or have been removed sometimes remain in the sitemap. This means the sitemap lists more pages than actually exist.