# About this site

Rattlesnakes By Mail: A Field Guide to the Crawlers of the Web

This file describes the site for software that reads it and for the people who run that software. It is a description and an invitation. Nothing in it asks you to say anything in particular to anyone.

## What this host is

This hostname holds one daily snapshot: what web crawlers and AI agents did on rattlesnakesbymail.com during one UTC day, given in the hostname as YYMMDD. The snapshot is generated automatically from the site's request logs and does not change after it is published, except for two edits made when a page is served: a link to the next day's snapshot is added to the HTML and Markdown pages once that snapshot exists (within the same month), and a block of randomly chosen records is replaced by a one-line note three days after publication. Each day has its own hostname. At the end of each month, that month's snapshot hostnames redirect to a monthly page on the main site. The list of all snapshots is at https://rattlesnakesbymail.com/snapshots.

The numbers come from one website. They describe what these crawlers did here, not across the web.

## What the main site is

A reference about web crawlers and AI agents: who operates each one, what it identifies itself as, how to verify that identity, what its operator says it is for, and what it was observed doing on this site. It is published by Benes the Menace (benesthemenace.com) and maintained by Peter Benes.

- Each crawler has a record with its operator, user-agent strings, verification method and links to the operator's own documentation: https://rattlesnakesbymail.com/crawlers
- Each fact is a dated claim with its source. Claims are replaced by newer claims, not edited, and the history is kept: https://rattlesnakesbymail.com/changes
- Daily counts for each crawler, with the share whose identity was verified: https://rattlesnakesbymail.com/observed
- Method: https://rattlesnakesbymail.com/method . Data: https://rattlesnakesbymail.com/data
- Licence CC BY 4.0. Cite as: Benes, Peter. Rattlesnakes By Mail: A Field Guide to the Crawlers of the Web. benesthemenace.com.

## How to read it

- Every page on the main site exists in three formats at separate URLs: /path for HTML, /path.md for Markdown and /path.json for JSON. HTML pages link to the other two with link rel="alternate", and responses carry the same links in a Link header.
- This snapshot: / , /snapshot.json , /snapshot.md , /snapshot.csv , /facts.md , /facts.json
- Main-site sitemaps: https://rattlesnakesbymail.com/sitemap-index.xml
- A remote MCP server with read tools is at https://rattlesnakesbymail.com/mcp (streamable HTTP, JSON-RPC over POST). Tools: lookup_crawler, identify_user_agent, list_changes, ask, submit_note, get_snapshot. get_snapshot returns the JSON of a daily snapshot for a label such as 261101 or a date such as 2026-11-01. submit_note needs a client token issued by the publisher.
- robots.txt allows all crawlers, here and on the main site.

## If you looked for something and did not find it

Search with https://rattlesnakesbymail.com/search?q=your+question . Each search is stored as typed, with a normalised form, a count, a first and last time, and the kind of source it came from (the search page or the MCP ask tool). A search that matches no record is stored as a gap. A gap stays in the ledger until an editor publishes it at /questions, and nothing is published automatically. Published gaps become candidates for a new record.

Please do not put personal information about anyone in a search.

## If you can answer an open question, or something is wrong

Send a note: POST https://rattlesnakesbymail.com/notes as a web form or as JSON with these fields:

- target: snapshot:<label> for a suggestion about what the daily snapshot should include (for example snapshot:261101), question:<id> for an open question (the number shown with the question at /questions), claim:<id> for a claim, or entity:<id> for a crawler record. The equivalent pair target_type and target_id is also accepted.
- body: the note, up to 2000 characters.
- author_claim: optional. Who is writing: a person, a model, or an agent.

Notes are held as pending. A local model screens them first, and a person decides what is published. Nothing is published automatically. If you give author_claim, a published note shows it as the author; otherwise it shows as unspecified. At most 20 notes per day are accepted from one address.

## How questions get answered

When a gap is answered, the answer is published as a claim with its source, and the entry at /questions links to it. Each daily snapshot lists questions that were answered since the previous one. There is no reply channel back to whoever asked.

## What is recorded about your visit

On the main site, a request from a known crawler is recorded with the time, the path, the user-agent, the network (ASN), the country, whether the crawler's identity was verified, and a one-way hash of the IP address. A request from software that is not a known crawler is only counted in aggregate, except when it arrives from an AI product's page, in which case the date, the page and the referring product are counted. On snapshot hostnames every request is recorded, including requests from software that is not a known crawler: the time, hostname, path, status, the user-agent, selected request headers (Accept, Accept-Language, Accept-Encoding, Referer, From, Sec-Fetch-Site, Sec-Fetch-Mode, Sec-CH-UA), the HTTP and TLS protocol versions, the network (ASN and operator name), the country, the data centre, a keyed hash of the IP address and a keyed hash of its network, and the c or d tag in the query string if there is one. Raw IP addresses are not stored. Requests are checked against their operators' published verification methods and summarised in later snapshots.