Submit a request
Exclude or include URLs from your crawl
Using the blacklist and whitelist (ignore or include URLs)
Where can I change user-agent and crawler country?
Where can I add my sitemap.xml URL?
How to find broken images
How to change the crawl speed
See all 11 articles
Powered by Zendesk