Exclude or include URLs from your crawl

If you need to have a closer look at a specific part of your domain or if you want to exclude some areas you can follow these methods:


Open the "project settings" in the top right corner, move to tab "Advanced Crawl"


1. Subfolder only


With this settings your crawl will only contain all data from a certain subfolder ( e.g. "/wiki/)


2. Blacklist/Whitelist


You can exclude URLs from your crawl by adding blacklist rules, in this example we want to exclude the Magazine and our Wiki, we realize that by blacklisting the subfolders. The rules should look like this:


This rule will exclude all URLs containing  /wiki/, please note that this does not take the folder hierarchy into account. For instance, a URL domain.com/wiki/ will be excluded as well as domain.com/subfolder/wiki/

You can apply any rule in any depth if you wish that certain sites be excluded e.g.:



You can add as many rules as you like.


Let's move on to the whitelist, the whitelist has the same functionalities as the blacklist but it works the opposite way. If you need to apply rules by "analyze only" then you can use our whitelist feature.


In this example, we want to crawl ONLY our Magazine and Wiki. We realize that by whitelisting each subfolder:



Please note that this will also whitelist domain.com/subfolder/wiki/, you may need to adjust it in depth (regex:https://en.ryte.com/wiki/)



3.Test your settings

It can be very time consuming if you run your rules without testing only to see that it did not work when the crawl finished. Please test your settings first, in order to get familiar with the syntax.


Here we are testing our whitelist rules from above, so only URLs from either the Wiki or the Magazine should be crawled.

If we enter a URL that should be excluded and the response status is 9xx (in this example 950) our settings are fine if the status is still 200 then the rules did not work.

We can also test in the other direction by test-crawling the included content by entering a URL within the whitelist rules:


If we applied the rules successfully then the test will respond with a status of 200, this only makes sense if we tested excluded URLs first!

Test settings Status Codes:

200 - OK

950 - blocked by whitelist

951 - blocked by blacklist


ImportantIf your test was not successful and you run it again with the same Test-URL it might be cached, please enter a different URL each time you run a test in order to not have cache discrepancy.

You can apply all rules at once (whitelist/blacklist, subfolder, subdomain etc) Please make sure that they don't cancel out each other!




Have more questions? Submit a request


Please sign in to leave a comment.
Powered by Zendesk