Crawl Policy
As of September 7, 2026
What this crawler does
Readings holds the products the AgeTech Prism™ has read from their public web footprint. Reading a product means fetching the pages the product's own site publishes, and the public record around it: app-store and retail listings, review and news sites, and public registries such as device authorizations, trial registers and recall databases. The placement rests on the same public record anyone else can check. This page states the policy every automated request follows, whichever site it reaches.
Identifying itself
Every request carries one identifying user agent string:
AgeTechPrismReader/0.2 (+https://www.agetechprism.org/crawler)The string resolves to this page, so a site operator who sees it in their logs can read the policy that governed the request.
robots.txt, rate limits
- robots.txt is honored. A site's robots.txt is read before any other page on it, and every path it disallows for this user agent (or for all crawlers, when it names no crawler specifically) is skipped.
- Crawl delay and a per-host rate limit. Requests to the same site are spaced at least one second apart, or by the site's own declared crawl delay when it sets a longer one. One request is in flight per host at a time, a bounded number of hosts are read at once, and each site is capped at a fixed number of pages per reading.
- Public registries are read through their public APIs (for example openFDA, ClinicalTrials.gov and PubMed), under those services' own terms and rate limits.
- A maker's site is read when a product is identified as a relabeled or resold version of that maker's product, so that the reading describes the device the buyer receives.
What it never does
Paywalls, logins, and anti-bot measures are never bypassed. A page behind one of those is a page this crawler does not read, and the reading it contributes to states that limit. The crawler may execute a page's own scripts to read the text a browser would show; it never does so to get past an access control.
What captured text is used for
Captured text is used to read a product from its public footprint: pricing, claims, privacy and terms language, manuals, listings, registry records, and similar public evidence, matched against the instrument the AgeTech Prism™ publishes. Publicly published video and images on a product's site may be transcribed or read for their text locally, on the machine that runs the reading, with a timestamp or image locator kept. The captured text is stored, alongside its source URL and capture date, so a reading's placements stay verifiable against exactly what was read.
Contact
Questions about a specific request, or a request to stop reading a particular site, go to info@agetechprism.org.