How to crawl a single folder?

RYTE supports two options for folder crawling.

Option 1 for beginners:
The shortest and easiest way to crawl a single folder on your domain is to open the Crawler-Settings on the top right of RYTE. There you can find many options. Navigate to "What to crawl" and choose "Subfolder-Mode". Here you can add your folder. Keep in mind that your folder must be relative to your root directory!

An example would look like this: you want to crawl your folder "awesome". Your URL looks like this: www.domain.com/awesome/. To crawl just the contents located in the awesome folder enter /awesome/ in the subfolder field. This is the instruction for the crawler to operate only in the folder /awesome/ and subfolders of /awesome/.

If you activate the subfolder crawling it is absolutely necessary to edit the homepage URL of the project. Click "homepage URL" by staying in the tab "what to crawl". It is important that the homepage URL contains the directory of the subfolder. If not - the crawl won't be successful. Please check the settings by clicking the tab "Test settings".

Option 2 for advanced users:
The expert option can also be found within the "Crawler-Settings". There is another tab named "URL Patterns". With the two options "Exclude URLs" and "Must include URLs" is a very precise crawling possible.

Exclude URLs:
URLs which contain an expression that was defined before a crawl will be excluded. For example: when /awesome/ is excluded all URLs containing /awesome/ will be ignored

Must include URLs:
URLs which contain an expression that was defined before a crawl will be included. For example: when /awesome/ is included the crawler is working only in this folder.

This feature is really powerful and does support regex. Examples can be found in the RYTE interface.

As already mentioned above. If you active the subfolder crawling it is absolutely necessary to edit the homepage URL of the project. Click "homepage URL" by staying in the tab "what to crawl". It is important that the homepage URL contains the directory of the subfolder. If not - the crawl won't be successful. Please check the settings by clicking the tab "Test settings".

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk