Crawl Policy
As of September 4, 2026
What this crawler does
Readings holds the products the AgeTech Prism™ has read from their public web footprint. Reading a product means fetching pages the product's own site publishes, so the placement rests on the same public record anyone else can check. This page states the policy every automated request follows.
Identifying itself
Every request carries one identifying user agent string:
AgeTechPrismReader/0.1 (+https://www.agetechprism.org/crawler)The string resolves to this page, so a site operator who sees it in their logs can read the policy that governed the request.
robots.txt, rate limits
- robots.txt is honored. A site's robots.txt is read before any other page on it, and every path it disallows for this user agent (or for all crawlers, when it names no crawler specifically) is skipped.
- Crawl delay and a per-host rate limit. Requests to the same site are spaced at least one second apart, or by the site's own declared crawl delay when it sets a longer one.
What it never does
Paywalls, logins, and anti-bot measures are never bypassed. A page behind one of those is a page this crawler does not read; the reading it contributes to states that limit rather than working around it.
What captured text is used for
Captured text is used to read a product from its public footprint: pricing, claims, privacy and terms language, and similar public evidence, matched against the instrument the AgeTech Prism™ publishes. The captured text is stored, alongside its source URL and capture date, so a reading's placements stay verifiable against exactly what was read.
Contact
Questions about a specific request, or a request to stop reading a particular site, go to info@agetechprism.org.