Repository navigation
Discoverability audits see the browser's view, not a plain HTTP agent's #17441
frmoretto
started this conversation in
Agentic Web Audits
Replies: 0 comments
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Uh oh!
There was an error while loading. Please reload this page.
Two observations from testing research-identifier sites (ORCID, ROR, Zenodo, Wikidata, OpenAlex) on 2026-10-25:
The llms-txt gatherer fetches /llms.txt via Network.loadNetworkResource with includeCredentials: true on the page frame, so it inherits the browser session, including any bot-challenge clearance.
On ror.org, every path (record pages, /robots.txt, /llms.txt) returns a 429 Vercel Security Checkpoint to non-browser clients, while the API host returns 200. An audit run in Chrome can report a reachable file that agents fetching over plain HTTP never get.
Would an informational check that repeats the fetch without credentials, and warns when the status differs, be in scope?
The ARD gatherer already parses the main document's Link header. The same pass could look for a declared structured alternate: Link rel="alternate" with a JSON or JSON-LD type, FAIR Signposting (describedby, cite-as), or an inline JSON-LD block.
Wikidata, OpenAlex and Zenodo would pass; orcid.org would not (it serves an identical 67 KB shell for every iD while content negotiation already returns JSON-LD; see Declare the JSON-LD alternate on public record pages ORCID/orcid-angular#2934).
Happy to share the curl commands, or to send a PR for (2) if it fits the category's direction.
All reactions