About our crawler

takumi_audit audits online stores at the request of people who want to improve them. If you saw our crawler in your logs, this page explains what it does.

How it identifies itself

Our HTTP crawler uses the user agent takumi_audit-bot/1.0 (+https://scan.takumi.agency/bot). Browser-based tests use a standard Chrome user agent with takumi_audit/1.0 appended, the same way Lighthouse marks its visits.

What it does

• Reads public pages only, at about two requests per second and at most two at a time.
• Obeys robots.txt, including Crawl-delay, and stops at a page cap.
• Backs off when your server answers 429 (Too Many Requests) or 503 with Retry-After: it pauses (as long as Retry-After asks, up to 30 seconds) and halves its rate, easing back only after a long run of normal answers.
• Visits pages in a real browser like one shopper would: browsing, searching, and adding one product to the cart.
• Runs passive checks of security headers, TLS certificates and DNS records.

What it never does

• Log in, create accounts, submit forms, enter personal or payment details, or place orders.
• Bypass bot protection or solve CAPTCHAs. If we're blocked, we stop and say so in the report.
• Scan for vulnerabilities, scan ports, fuzz inputs or load-test your site.

Blocking or allowing us

To block the HTTP crawler, add a robots.txt group for takumi_audit-bot. To allow audits you requested through bot protection, allowlist the user agents above. Questions? Email bot@example.com.