How to increase crawl speed

The crawl speed can be adjusted in the project settings by changing the number of parallel requests during the crawl. We usually crawl with 1-10 parallel requests but you can increase the amount by verifying your domain.

 

Changing the number of parallel requests

First, navigate to the project settings. On the project setup page, you can see the slider that shows the number of parallel requests. Drag the blue bar to change the number of parallel requests.

 

You can also adjust the slider in the project settings under "Advanced analysis" > "How to analyze" > "How fast should the analysis be?" You can also set up a wait time for the crawler between requests, further adjusting the speed of your crawls. Per default, there is no waiting time between requests.

 

 

Verifying your domain 

If you would like to crawl with more than 10 parallel requests you first need to verify your domain. To do that you need to upload a specific file to the root directory of your domain. Once the file is uploaded click on “Check authentication”. Once we detect the file on your domain you can increase the number of parallel requests.

Please be aware that a higher amount of requests puts the server under more strain. If you’re unsure how many requests your server can handle we recommend increasing the crawl speed only slowly as to avoid server timeouts.

 

Have more questions? Submit a request

0 Comments

Please sign in to leave a comment.
Powered by Zendesk