Why Snaglist always follows robots.txt, and how you see this in the report.
Snaglist always follows robots.txt. This is fixed and cannot be disabled. We skip any directory that is blocked for crawlers.
If you want Snaglist to include a directory, allow it in robots.txt. Only do this if Google is also permitted to access it.
The export lists the URLs we skipped, marked with "Blocked by robots.txt".