Can I whitelisten the user agent via Wordfence?

WordFence for WordPress is a big topic at RYTE. We are looking for a solution.
We are using dynamic IPs. They are changing very fast and it's possible to crawl your website with the same Useragent but with different IP-addresses.

The RYTE Crawler is using the user-agent "OnpageBot" by default.

Unfortunately, there is no way to whitelist an user-agent in WordFence, only IP addresses. The number of our crawler IP-addresses is enormous, the IP addresses itself are changing constantly.

One possible whitelisting solution is to whitelist the whole IP-range of Amazon Webservice (AWS) Europe in Ireland. But this would open the doors for many other crawlers as well.

You can deactivate the blocking of fake google crawlers in WordFence, but just for a few hours until we have finished crawling your domain and turn it on again afterwards.

Once we provide a solution, which we are currently working on, we will keep you updated here.

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk