Crawls & logs

What it's for

The crawl history and per-crawl log: what was fetched, status codes, redirects, blocked/again-skipped URLs and why a crawl stopped. Your audit trail when results look surprising.

When to use it

• When a crawl found fewer pages than expected.
• To diagnose robots.txt blocks, redirect chains or server errors.

How to use it

1. Project → «Logs» (or open a crawl from the history).
2. Filter by status code to find 4xx/5xx and redirects.
3. Cross-check a surprising URL against its fetched status.

Reading your results

For each output: is it good, does it need improvement, or is it bad — and exactly what to do next.

Next steps

• After fixing redirects/errors, re-crawl and confirm they're gone.
• If pages are missing, check robots.txt and internal links to them.