RYTE and Yoast say NoIndex via robots.txt

"Do you by any chance use bot blockers or bot blocking plugins? Like e.g. Wordfence?

One common reason, why domains are not indexable according to RYTE are plugins differentiate between bot visitors and browser visitors.

So when you open the page with a crawler system you would get the feedback """"<META NAME=""""robots"""" CONTENT=""""noindex,nofollow""""> """" even though when opening your page in a browser the reply will be""""<meta name=""""robots"""" content=""""noodp""""/> """".

A possible explanation would be that your server replies different versions of the page depending on the User-Agent or IP Address.

Please make sure that our crawler is allowed to crawl the website. Of course, you can whitelist our crawlers User-Agent which is ""OnPageBot"".

Here is an interesting link about Common SEO Mistakes and How You Can Fix Them."

Have more questions? Submit a request


Please sign in to leave a comment.
Powered by Zendesk