WebtelemetryBot

WebtelemetryBot is the crawler behind webtelemetry.dev — a background process that discovers newly-launched websites and records basic, passively-observable health and security signals (TLS configuration, security headers, SPF/DMARC, general tech stack, title and meta tags).

What it does and does not do

• Fetches your homepage over HTTP(S), the same as any browser would.
• Reads your robots.txt before crawling anything and honors it.
• Makes at most one request to your site at a time, with a minimum delay between requests.
• Never attempts login, port scanning, directory brute-forcing, or any other active/intrusive check unless you have explicitly claimed and verified ownership of your site through webtelemetry.dev.
• Does not store your page's HTML or JavaScript long-term — only extracted, compact metrics.

User-Agent

WebtelemetryBot/1.0 (+https://webtelemetry.dev/bot)

How to block it

Add the following to your site's robots.txt to disallow all crawling by WebtelemetryBot:

User-agent: WebtelemetryBot
Disallow: /

We check robots.txt before every crawl and will stop visiting within one crawl cycle.

Questions or abuse reports

If you believe this bot is misbehaving, crawling too aggressively, or you have any other concern, contact fedja.hadziselimovic@gmail.com.