The three most common causes we see, and what you can do about them.
If a crawl stops much earlier than you expect, it’s almost always one of these three.
The most common cause. The first pages go fine, then the site rejects us with status 403 or 429. After ten consecutive rejections, Snaglist stops to avoid overloading the site.
Whitelist SnaglistBot in your firewall. How to do this is explained in Whitelist SnaglistBot.
Per crawl, Snaglist fetches a maximum of 2,000 pages with Solo, and 10,000 with Studio and Agency. If the site is larger, the crawl stops at that limit.
Snaglist finds pages by following links and by reading the sitemap. A page with no links pointing to it and not listed in the sitemap won’t be found by any crawler. Not even Google.
Can’t figure it out? Ask your question directly from the crawl using the "Question about this crawl" button. We’ll see exactly what happened.