Where can I change user-agent and crawler country?

In case the crawl did not get any data or seems to be crawling for a long time without finding any URLs a change of the useragent or crawler country can sometimes help.  

To check whether our crawler is able to crawl your domain and find any URLs in the current crawl have a look at the crawl history: (Project Settings -> Crawl history )UA2.pngIn case there are 0 URLs found after a decent amount of time into the crawl you can cancel the crawl and retry with a different User-agent and/or Crawler country:


The option "Crawler country" will change the server location from where the crawler will analyze.

The option user-agent will determine the name of the crawler, you can enter firefox as user-agent for example. Or if you want to make sure that you whitelist our service only you can give the crawler an individual name (e.g:crawler123abc) 


This option is only available with Pro account or higher!


Have more questions? Submit a request


Please sign in to leave a comment.
Powered by Zendesk