I have many URLs with parameters in my Ryte reports. They are generating many duplicate content pages. Which steps are required to clean up my Ryte reports?

The most common e-commerce solutions are aware of URLs with parameters and that these URLs are not essential for the crawlers of search engines and for users.


Ideally, your e-commerce solution is supporting an option to add rules in the robots.txt file of your website. The robots.txt sets up rules for User-agents (crawler are using User-agents) and allows or disallows them to crawl specific URLs or folders.

When you already have a fully operating robots.txt you can click on "Crawler-Settings" in the upper right corner of Ryte choose "robots.txt behaviour" in the tab "How to Crawl". Now you can choose between crawl everything on your domain or crawl according to the robots.txt on your server.

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk