V1 surface
V1 observes only a site's public /robots.txt. It records the HTTP/source state, a source hash, the matching relevant directives, parser version and taxonomy version. Permanent report GETs are database reads and never re-fetch the target.
Interpretation
Known agents are evaluated against exact agent groups, then wildcard fallback. Results are normalized as ALLOW_ALL, BLOCK_ALL, PARTIAL, NO_RESTRICTION_DECLARED or UNDETERMINED. Content-Signal values are kept separately as declared AI-use signals.
Trust boundary
A declared robots policy is not proof that a provider crawled, did not crawl, complied, or was technically prevented from accessing content. User-directed agents and policy tokens are labeled separately where provider semantics differ.
Safety
Observation accepts public domain names only. Private/reserved destinations are rejected, redirects are revalidated, ports are limited to 80/443, response size and time are bounded, and public observation requests are rate-limited.