100 top websites fail agent readiness test
Ashley Pfeiffer scored 100 of the web's most visited sites on agent readiness using a four-dimension framework (Discovery, Structure, Actions, Policies) and found all failed, with an average score of 40 out of 100, prompting the creation of agent.json—an open spec for websites to declare capabilities and policies to AI agents.
Score breakdown
Developers and site operators can use agent.json and the agentweb toolkit to make their websites discoverable and safe for AI agents to interact with, closing a critical gap in how the web currently supports agent-driven interactions.
- 01Ashley Pfeiffer scored 100 of the web's most visited sites on agent readiness across four dimensions (Discovery, Structure, Actions, Policies) with 25 points each, finding an average score of 40, a maximum of 48, and zero passing sites
- 02Most sites score well on Structure (semantic HTML, Schema.org markup, sitemaps) but fail entirely on Discovery and Policies—no site declares machine-readable capabilities or brand voice guidelines for agents
- 03agent.json is an open specification that sits at a domain root like robots.txt and declares capabilities, brand voice, rate limits, data handling policies, and human escalation triggers in JSON format
The web was built for human readers and search engine crawlers, but AI agents—Claude, ChatGPT, Siri, Copilot—now need to interact with websites on behalf of users. Pfeiffer developed a scoring framework across four dimensions (25 points each): Discovery (machine-readable capability manifests, MCP endpoints, llms.txt files), Structure (semantic organization, Schema.org markup, clean HTML, sitemaps), Actions (public APIs, OpenAPI/GraphQL specs, documented authentication), and Policies (rate limits, brand voice guidelines, escalation paths, data handling rules). Testing 100 of the most visited sites revealed an average score of 40, a maximum of 48, and zero passing sites. The critical gap lies in Discovery and Policies—websites provide no machine-readable way for agents to learn what they can do, and no guidelines for how agents should represent the brand or when to escalate to humans.
To close this gap, Pfeiffer introduced agent.json, an open specification that declares capabilities, brand voice, policies, and human escalation triggers in a JSON file at the domain root. The toolkit includes `npx agentweb score` for evaluating sites, `npx agentweb init` for generating starter configurations by industry, and `@agentweb-dev/middleware` for proxying any website as an MCP server without modifying the origin. All packages are open source and published to npm. Pfeiffer, a Director of Client Delivery at Cherre (a real estate data platform), frames agent.json as a unified discovery layer for the open web—analogous to the data infrastructure work of unifying fragmented sources into a single source of truth.
Key facts
- 01Ashley Pfeiffer scored 100 of the web's most visited sites on agent readiness across four dimensions (Discovery, Structure, Actions, Policies) with 25 points each, finding an average score of 40, a maximum of 48, and zero passing sites
- 02Most sites score well on Structure (semantic HTML, Schema.org markup, sitemaps) but fail entirely on Discovery and Policies—no site declares machine-readable capabilities or brand voice guidelines for agents
- 03agent.json is an open specification that sits at a domain root like robots.txt and declares capabilities, brand voice, rate limits, data handling policies, and human escalation triggers in JSON format
- 04The agentweb toolkit includes CLI commands to score sites (`npx agentweb score`), generate starter configurations by industry (`npx agentweb init`), and proxy any website as an MCP server without code changes (`@agentweb-dev/middleware`)
- 05All packages are open source and published to npm, with a full scoring report and repository available at github.com/ashpfeif12/agentweb