Why can´t you give out the IPs you are using to crawl my website?

We can not tell you the IP addresses we are using. Firstly because I do not know them and secondly for the reason that we are using hosts where we are not the only clients. So if you would whitelist all the IP addresses that we could possibly use you would also whitelist all other services that are hosted within our datacenter.

Therefor we offer the solution to whitelist our User-Agent: "OnPageBot", this can usually be whitelisted on your server and maybe Sitelock also offers a fix for this.

Alternatively you could define a static cookie that we could define in our crawler settings, so that every visitor with this cookie gets whitelisted.

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk