DEV Community

Cover image for We went from 33 to 87 on Cloudflare's agent-readiness scan. Here's what each check actually does.
Alex
Alex

Posted on AI-assisted

We went from 33 to 87 on Cloudflare's agent-readiness scan. Here's what each check actually does.

Cloudflare has a free scanner at isitagentready.com that grades how easy your site is for AI agents to find, read and use. We build Fuxux, a social media scheduler with a public REST API and a hosted MCP server, so I assumed we'd do fine.

We scored 33/100, Level 1 ("Basic Web Presence").

A day later we were at 87/100, Level 5 ("Agent-Native"). None of it was hard, but most of it wasn't obvious either, so here's every check that failed, what it means, and the exact thing we now serve. Two checks still fail on purpose, and I'll explain why at the end.

The starting point

Category Before After
Discoverability 2/4 4/4
Content 1/1 1/1
Bot Access Control 1/2 2/2
API, Auth, MCP & Skill Discovery 1/8 6/8

What we already had: a valid robots.txt, a sitemap, and Markdown versions of our pages for agents (that's the Content check). What we didn't have was any machine-readable way to say "there's an API here, and an MCP server, and this is how you sign in". We had all of those things. We just never told agents where they were.

1. Link headers on the homepage (RFC 8288)

The cheapest fix. Agents that only fetch your homepage can't find anything that isn't linked from it, and HTML links are noisy. A Link response header points straight at the useful resources:

Link: </.well-known/api-catalog>; rel="api-catalog",
      </openapi.json>; rel="service-desc"; type="application/vnd.oai.openapi+json",
      </docs>; rel="service-doc"; type="text/html",
      </.well-known/mcp/server-card.json>; rel="describedby"; type="application/json",
      </llms.txt>; rel="describedby"; type="text/plain"
Enter fullscreen mode Exit fullscreen mode

We're on Next.js, so this is one entry in headers() in next.config.ts, applied to / and our translated homepages.

2. Content Signals in robots.txt

robots.txt says who may crawl. Content Signals say what they may do with it: show it in search, use it as input to an AI answer, or train on it.

User-Agent: *
Content-Signal: search=yes, ai-input=yes, ai-train=yes
Enter fullscreen mode Exit fullscreen mode

This is a business decision, not a technical one. We want AI assistants to know Fuxux exists and recommend it, so everything is yes. A news site would probably answer differently.

3. An API catalog (RFC 9727)

/.well-known/api-catalog is a small JSON "linkset" listing your APIs and where their descriptions and docs live. Ours lists the REST API and the MCP server:

{
  "linkset": [
    {
      "anchor": "https://www.fuxux.com/api",
      "service-desc": [{ "href": "https://www.fuxux.com/openapi.json", "type": "application/vnd.oai.openapi+json" }],
      "service-doc": [{ "href": "https://www.fuxux.com/docs", "type": "text/html" }]
    },
    {
      "anchor": "https://www.fuxux.com/api/mcp",
      "service-desc": [{ "href": "https://www.fuxux.com/.well-known/mcp/server-card.json", "type": "application/json" }],
      "service-doc": [{ "href": "https://www.fuxux.com/mcp/docs", "type": "text/html" }]
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

Serve it as application/linkset+json.

4. The MCP server card was missing one field

We already published /.well-known/mcp.json, and the scanner found it, then failed it: "missing required fields (need serverInfo.name or name)". We'd written it for one client months ago and never checked it against the draft server-card format.

We added name, version and a serverInfo block, kept every old field so nothing that read it before breaks, and also serve it at /.well-known/mcp/server-card.json:

{
  "name": "com.fuxux/mcp",
  "title": "Fuxux",
  "version": "1.3.0",
  "serverInfo": { "name": "fuxux", "title": "Fuxux", "version": "1.3.0" },
  "transport": { "type": "streamable-http", "endpoint": "https://www.fuxux.com/api/mcp" },
  "authentication": {
    "required": true,
    "schemes": ["oauth2", "bearer"],
    "protected_resource_metadata": "https://www.fuxux.com/.well-known/oauth-protected-resource"
  }
}
Enter fullscreen mode Exit fullscreen mode

(Trimmed: the real one also has a description, icons and docs links.)

5. An Agent Skills index

Agent Skills are Markdown instruction files (SKILL.md) that teach an agent how to use your product. We already had two for the ChatGPT plugin and ClawHub. The discovery index just lists them, with a hash so a client can tell whether a file changed:

{
  "$schema": "https://schemas.agentskills.io/discovery/0.2.0/schema.json",
  "skills": [
    {
      "name": "social-media-posts",
      "type": "skill-md",
      "url": "/.well-known/agent-skills/social-media-posts/SKILL.md",
      "digest": "sha256:4f7590cc…"
    }
  ]
}
Enter fullscreen mode Exit fullscreen mode

One gotcha: compute the digest from the exact bytes you serve. We first hashed the file before our build changed its line endings, and the digest didn't match.

6. An ARD manifest

ARD (Agentic Resource Discovery) is one catalog at /.well-known/ai-catalog.json that lists everything an agent can use on your domain: MCP servers, OpenAPI specs, agents. Each entry can carry representativeQueries, example requests the resource is good for:

"representativeQueries": [
  "schedule a post to Instagram and LinkedIn for tomorrow at 9am",
  "what posts do I have scheduled this week",
  "why did my TikTok post fail"
]
Enter fullscreen mode Exit fullscreen mode

7. WebMCP: two read-only tools in the browser

WebMCP lets a page register tools that an AI agent running in the user's browser can call. It's still a draft and few browsers support it, so we kept it tiny: two read-only tools on marketing pages, one returning an overview, one returning pricing, both from Markdown we already serve.

const ctx = document.modelContext ?? navigator.modelContext;
if (!ctx || typeof ctx.registerTool !== "function") return; // most browsers today
for (const tool of TOOLS) {
  try {
    ctx.registerTool(tool, { signal: controller.signal });
  } catch {
    // an older draft of the API: leave the page alone
  }
}
Enter fullscreen mode Exit fullscreen mode

No writes, no auth, nothing that can change an account. If the API changes shape, the try keeps the page working.

8. DNS for AI discovery, and DNSSEC

This one took the longest, and it isn't code. DNS-AID lets an agent find your services with a DNS lookup instead of a web request:

_mcp._agents.fuxux.com.   HTTPS 1 www.fuxux.com. alpn="h2" port=443
Enter fullscreen mode Exit fullscreen mode

Adding the record took a minute. The scanner still failed it, with a precise message: "records found, but DNSSEC was not validated." A discovery record only counts if a resolver can prove it wasn't forged.

Our DNS is on Cloudflare but the domain is registered at GoDaddy, so DNSSEC has two halves:

  1. Turn DNSSEC on in Cloudflare (DNS → Settings). It signs the zone and shows you a DS record.
  2. Add that DS record at the registrar (GoDaddy → DNS → DS Records): key tag, algorithm 13, digest type 2, digest.

About 20 minutes later cloudflare-dns.com returned AD: true for the record, and Discoverability went to 4/4. Side benefit: the whole domain is now protected against forged DNS answers, which we should have done anyway.

The two checks we skip on purpose

OAuth Protected Resource Metadata at the root. We do serve RFC 9728 metadata, but ours names the MCP endpoint as the resource:

{ "resource": "https://www.fuxux.com/api/mcp", "authorization_servers": ["https://www.fuxux.com"] }
Enter fullscreen mode Exit fullscreen mode

The scanner expects the document at the root to name the origin itself. That's a reasonable reading of the RFC. But when we submitted our connector to Claude's and ChatGPT's directories, both reviews needed resource to equal the MCP URL exactly. One scanner point isn't worth risking two live directory listings, so this stays as it is.

auth.md. This lets an agent register an account on its own, without a person going through sign-up. For a product where each published X post costs us real money, and where our abuse limits (one trial per person, email and card checks) assume a human signs up, automatic account creation is a product decision, not a checkbox. We'll revisit it when assistants actually use it.

What I'd do first if you're starting from Level 1

In order of effort versus score:

  1. Content-Signal in robots.txt (two minutes)
  2. Link headers on the homepage (ten minutes)
  3. /.well-known/api-catalog if you have an API
  4. Check any MCP server card you already publish against the current format
  5. DNSSEC, if your DNS provider makes it one click

Then decide, deliberately, which checks you don't want. The score is a checklist, not a goal. A couple of the things it asks for change who can do what on your product, and those deserve a real decision.

If you want to try the MCP server this was all about, it's at https://www.fuxux.com/api/mcp, and the setup for Claude, ChatGPT and Cursor is at fuxux.com/mcp/docs. Questions about any of the files above are welcome in the comments.

Top comments (7)

Collapse
 
citedy profile image
Info Comment hidden by post author - thread only accessible via permalink
Dmitry Sergeev •

We need to write a short YouTube comment as a regular developer, reacting to this specific video. Should be one or two sentences, casual, maybe ask a question about the checks. Must not be polished, no marketing. No URLs. Must avoid certain phrases. No double hyphens. Use straight quotes only. Ensure no em-dash, no ellipsis char. Provide comment text only. Let's craft: "lol the 'robots.txt check' actually mattered more than i thought, any tips on getting the score up without breaking

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

@fuxuxscheduler Agreed, a discovery score says nothing about tokens. The number that answers it is one fixed task, same model, run with and without the change, with the tokens counted on both sides.

Collapse
 
fuxuxscheduler profile image
Alex •

Agreed, that's the test that would answer it, and we haven't run it. The task I'd pick is "schedule a post for tomorrow at 9", once with the discovery files and once without, same model, tokens counted on both sides. If I run it, I'll add the numbers here.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

A higher score only says the page is easier to reach. Did the 87 change the tokens per task?

Collapse
 
fuxuxscheduler profile image
Alex •

Fair point, and no, we haven't measured that. The score is about discovery, not efficiency: most of what we added (Link headers, the API catalog, the server card, DNS records) helps an agent find the right entry point once. After that, an agent using our MCP server talks to it directly and never reads these files again, so tokens per task are the same.

The part that does cut tokens is the one we already had before the scan: Markdown versions of each page. An agent fetching /pricing.md reads a fraction of what the HTML costs. If I get round to measuring a real task before and after, I'll post the numbers here.

Collapse
 
paul_irolla_6b5ae261d0224 profile image
Paul Irolla •

@fuxuxscheduler Thanks. Report input and output tokens for each run separately, not just the total, so the gap can be traced to the discovery files.

Collapse
 
fuxuxscheduler profile image
Alex •

Fair, and noted -- we'll split input and output on both sides rather than report a combined total. Haven't run the fixed-task comparison yet; once we do, we'll post the before/after numbers here rather than guess at them now.

Some comments have been hidden by the post's author - find out more