Skip to content
Webitops

Notes

Making a public data site legible to AI agents

MCP servers, llms.txt, RFC 9727 API catalogues and well-known discovery documents — what we actually shipped to make a public data service consumable by machines, and which parts earned their keep.

5 min read MCP, Discoverability, Laravel

There is a reasonable chance that a meaningful share of your site’s readers are no longer people.

Not crawlers indexing you for a results page — that has been true for thirty years — but agents retrieving your content to answer a question right now, on behalf of someone who will never see your page. If your site publishes data, that audience needs something other than HTML, and the usual arrangement is unpleasant for everyone: they scrape, you get load and no attribution, and they get a number with no provenance.

We had a service where this was worth solving properly, and no commercial reason to keep the data scarce. Here is what we shipped, in the order it mattered.

1. A machine-readable answer to the actual question

Start here, because everything else is signposting to it.

Ours was an MCP server: stateless JSON-RPC 2.0 over Streamable HTTP, exposing a single tool that answers the one question the site exists to answer, in any requested currency, with the timestamp and source of the underlying data attached.

A few things we would tell anyone building one:

  • Stateless is the right default. Streamable HTTP without a session layer means the server is an ordinary rate-limited HTTP endpoint, deployable and scalable exactly like the rest of the app. Sessions buy you very little for a read-only tool and cost you a lot of operational surface.
  • Negotiate protocol versions explicitly, and support more than one. The specification is moving. Clients in the wild pin different revisions. Handle a version you do not know by responding with one you do, rather than failing.
  • One tool that answers the real question beats five that expose your schema. The temptation is to publish get_metals_price, get_fx_rate, list_currencies and let the agent assemble the answer. Do not. Every join you push onto the caller is a chance for them to compute something wrong and attribute it to you.
  • Return provenance in the payload. Not just the number — when it was computed, what it was derived from, what the caveats are. This is the single biggest advantage you have over being scraped, and it costs three extra fields.

2. Making it findable without being told

An endpoint nobody can discover is a private API with extra steps. This is the part that is genuinely new, and it is mostly cheap static documents.

  • /.well-known/mcp.json — a server card. An MCP client that knows only your domain can find the server, learn what it exposes, and connect.
  • /.well-known/api-catalog — the RFC 9727 linkset. A general-purpose way to say “here are my machine interfaces”, not specific to any one protocol, which matters because MCP will not be the last of these.
  • Link headers on ordinary responses, advertising the sitemap, the catalogue and llms.txt. Consumers that fetch a page and never parse the body still see them.
  • llms.txt and llms-full.txt — a plain-language map of the site, and a fuller dump for consumers that want everything in one request.

The principle underneath: a machine should be able to get from your bare domain name to a typed answer without a human reading your docs. Test that path end to end, because it is easy to ship four correct documents that do not actually chain.

3. Saying what you allow, on purpose

Most robots.txt files are ambiguous about AI use by accident — they were written before the question existed, and silence gets interpreted by whoever is doing the interpreting.

We name the major AI user agents explicitly and declare content signals separating indexing, inference-time retrieval and training. For this project all three are permitted, because the point of the service is to be used. That is not the right answer for everyone. The recommendation is not “allow everything” — it is decide, and write it down, because the alternative is having it decided for you.

What we would skip

Being honest about the parts that have not paid off yet:

  • llms-full.txt has seen little use. Cheap to generate, so it stays, but we would not build it first.
  • Discovery specifications are young. The RFC is stable; the MCP-adjacent conventions are not, and some of this will need revisiting. If you have limited time, ship the tool and the server card, and add the catalogue later.
  • Nobody has a good analytics story here. Agent traffic is hard to distinguish from ordinary API traffic, so “did this work?” remains partly a matter of faith. If you need a measurable return before you build, this is not yet that.

Was it worth it?

For a public-interest service where being used correctly is the goal, clearly yes — the marginal cost over an already-public JSON API was small, and it replaced being scraped badly with being consumed properly.

For a commercial product where the data is the moat, the calculus is entirely different, and the honest answer may be that you want the opposite of all of this.