The default settings of the crawler will generally allow the crawler to absolutely crush poorly optimized sites.

The thing to keep in mind is that most people that want the crawler are the ones whose pages take 5~10 seconds to load when cold / not cached.

By default the crawler is going to hit pages fast enough that they will stack up and exhaust all resources the account has.

Providers can adjust the limits / defaults so that it is more reasonable – such as 1 request per 15 or 30 seconds – to account for this but most don’t.

To be fair I’ve only seen about ~3 requests for the crawler since it was introduced. In all 3 occasions we enabled it for that user’s account but did not do so server-wide.

Source

Leave a Reply

Your email address will not be published. Required fields are marked *