BetterSaaS
ResolvedhighCrawlerProblem #13

Crawler ignores robots.txt crawl-delay

Rankforge

50% of problems resolved · median first response 34h

Our staging environment was crawled at 40 req/s despite a crawl-delay of 10, which triggered our rate limiter.

Environment

Web · Chrome 141

Response timeline

Every state change is public, including the company's reasoning.

  1. Reported

    Jul 11, 2026

    Sam Oduya reported this problem as a verified user.

Other Rankforge problems

AcknowledgedmediumContent

Content brief citations 404 after 30 days

Cited source URLs stop resolving after a month because the cached snapshot expires, which breaks the audit trail.

9 also affectedReported 12d agoResponded 10d ago