User-Agent Sniffing vs Content Negotiation: Why plym lets agent decide what they want

The short answer

plym no longer detects AI user agents. Markdown is opt-in by header. Ask for it with Accept: text/markdown and you get Markdown. Don't ask, and you get HTML — no matter who are you.

It's based on a simple engineering principle: stop profiling the client, start responding to what the visitor asked for.

Even though the agentic era is new, this design choice has been winning for last 30 years, through the browser wars & surge in mobile traffic.

Introduction

Many years ago, web applications would try to identify the browser, and then figure out if that browser + version supports a particular feature, and then decide what client would see.

This was known as browser sniffing.

Soon after that we learnt that there are several problems with this approach. We then switched to feature detection.

Instead of inspection, ask the browser if it supports a feature. The name or identity of the browser was irrelevant anyway.

Then the mobile traffic started seeing a surge. And we decided to do similar inspection, and then started serving them m-dot version of that application. It carried the same set of problems, and we soon started asking the client the width of the viewport.

Both of these changes followed the same engineering principle. Stop making assumptions on identity, ask what the client wants.

The new shift in traffic

Something similar has been happening with rise of agent traffic on the web. Agents don't care about the HTML soup. So why not inspect their user-agent and show them the content as markdown?

During early days of development, plym did something similar. It looked at the User-Agent of a reqeust. If the string matched one of forty-eight names on in a ai-crawlers.Caddy file, it handed back markdown. Everybody else got HTML. It worked. Crawlers got clean text, humans got a rendered page, and the token cost of reading a plym site dropped by something like three quarters.

It seemed like the obvious, smart move.

Then we deleted the file.

This post is the whole argument — the case for sniffing, the case against it, the thing that actually decided it for us, and what we ship today. If you run a docs site, a blog, or a CMS, you are about to face the same fork.

What user-agent sniffing looked like

Here's the config we shipped, docker/ai-crawlers.caddy:

# abridged — the real regex is one line and lists all 48 names
@llm `(header_regexp('User-Agent', '(?i)(gptbot|chatgpt-user|oai-searchbot|claudebot|
  claude-user|claude-web|claude-searchbot|claude-code|anthropic-ai|perplexitybot|
  perplexity-user|googleother|google-notebooklm|google-cloudvertexbot|
  gemini-deep-research|meta-externalagent|meta-externalfetcher|bytespider|ccbot|
  amazonbot|bedrockbot|cohere-ai|...)') || header({'Accept': '*text/markdown*'}))
  && file({'try_files': [path + '.md']})`

handle @llm {
    rewrite * {path}.md
    header Content-Type "text/markdown; charset=utf-8"
    header Vary "User-Agent, Accept"
    file_server
}

One regex, forty-eight names, a Markdown twin on disk for every published page. The Vary header was correct. The caching was correct. Nothing about it was sloppy.

It was still the wrong shape.

The case for sniffing (it's better than you think)

Let's steelman it properly, because the people arguing this side are not fools.

HTML is genuine overhead for a text pipeline. Fastly measured it plainly: a typical article "compresses to 20–30% of its HTML size" as Markdown. Every nav bar, cookie banner, and <div class="wrapper-inner-flex"> is tokens the model pays for and then throws away. Guides tracking Cloudflare's rollout put the reduction at 60–80%.

Nobody sends the header yet. This is the strongest pro-sniffing argument by a distance. Accept: text/markdown only works if the client sends it. Most crawlers, today, send */* or a copied browser Accept string. Sniff and you serve Markdown to everyone who wants it. Negotiate and you serve Markdown to the handful who know to ask.

The client stays in control, technically. Fastly's own recipe leads with user-agent matching against seventeen known crawlers, with Accept as a fallback and an /md/ path prefix for debugging. Their framing: it "respects the agent's workload, but also your bandwidth."

Content negotiation looks like sniffing anyway. Human browsers never send Accept: text/markdown. So in practice, only bots get the Markdown variant. If your objection to sniffing is bots see something humans don't, negotiation lands in the same place by a politer route.

That last one deserves a straight answer, and it gets one below.

The case against sniffing

It is cloaking under the plainest reading of the rules. Google's spam policies define cloaking as showing "different content or URLs to human users and search engines." Serving Markdown off a User-Agent match is exactly that, and the search teams have said so out loud. In February 2026, Google's John Mueller called converting pages to Markdown for LLMs "such a stupid idea," and asked the question that sticks: "Why would they want to see a page that no user sees?" Lily Ray put the policy angle bluntly — it "directly violates search engines' longstanding policies about this (basically cloaking)."

The bot-only version rots. Microsoft's Fabrice Canel made the operational argument, and it's the one that should scare you: "Non-user versions... are often neglected, broken. Humans eyes help fixing people and bot-viewed content." Nobody opens the Markdown twin on a Tuesday to check whether the code block still has its language tag. It breaks quietly, for months.

User-Agent is an unauthenticated string. Any curl can claim to be GPTBot. Real verification means forward-confirmed reverse DNS or checking published IP ranges — infrastructure work that a regex in a Caddyfile does not do. So a sniffing rule is a promise you serve special content to anyone who types the right word.

The list is a treadmill you never finish. Forty-eight names covered the agents that existed when we wrote it. Every new agent shipped unlisted and got HTML. Every renamed agent silently fell off. You are maintaining a registry of the entire AI industry inside a regex, forever, and you are always behind.

The argument that actually decided it

Look at our old list again. It contained googleother. It contained semrushbot-swa. It contained claude-code.

Ask what those three came to do.

GoogleOther is a general-purpose fetcher used across Google teams for one-off crawls and product work. SemrushBot-SWA is a site-audit crawler — it exists to look at your page. And Claude Code is usually building something against your site, not summarizing it.

We handed all three a stripped text file and congratulated ourselves on saving tokens.

That is the mistake, and it has a name.

A user agent tells you who. It never tells you why.

Anthropic publishes three distinct crawlers with three distinct jobs: ClaudeBot gathers content that may inform model training, Claude-User fetches a page because a person just asked a question, and Claude-SearchBot indexes for search quality. Three purposes, one vendor. OpenAI, Google, and Perplexity all split the same way.

And the fastest-growing category doesn't want your text at all.

• ChatGPT Atlas reads the page through ARIA tags — the same roles and labels that power screen readers.
• Playwright MCP hands models accessibility snapshots rather than screenshots.
• Claude's computer use works from pixels: screenshot, reason, act, repeat.
• Perplexity Comet runs a hybrid — accessibility tree plus selective vision.

Markdown has no ARIA. No roles. No focus order, no form controls, no landmarks, no layout. For an agent that came to click something, Markdown isn't a lighter version of your page. It's a page with the controls sawn off.

The four jobs framework

Here's the model we now use internally. Every agent request is doing one of four jobs:

Two of four want Markdown. Two of four are actively damaged by it.

A User-Agent string cannot distinguish between these four, because the same vendor — sometimes the same crawler — does more than one. An Accept header distinguishes between them perfectly, because the client declaring its intent is the signal.

That's the whole insight. Stop inferring intent. Let the client state it.

What plym does now

One rule, in docker/Caddyfile:

@md {
    header_regexp Accept (^|,)\s*text/markdown\s*(;[^,]*)?($|,)
}
handle @md {
    header Content-Type "text/markdown; charset=utf-8"
    header Vary "Accept"
    file_server
}

What that gives you:

1. Same URL, two representations. /blog/my-post returns HTML or Markdown depending on what you asked for. No .md twin URL to maintain, no separate sitemap, no second thing to keep in sync.
2. Vary: Accept, so CDNs and proxies cache the two variants separately instead of poisoning each other.
3. ClaudeBot with a browser Accept header gets HTML. No exceptions, no allowlist.
4. The Markdown is generated from the same source as the HTML at publish time, so it can't drift.

Try it against any plym site:

curl -H "Accept: text/markdown" https://plym.io/blog/some-post

And from our own docs, in What plym does not do: "plym does not detect AI user agents. A request from ClaudeBot/1.0 carrying a browser Accept header receives HTML."

Answers to the obvious doubts

Isn't content negotiation just cloaking with extra steps?

No, and the distinction is load-bearing. Cloaking is the server deciding what you get based on who it thinks you are. Content negotiation is the client deciding what it gets based on what it asked for. Same URL, both cases, but the direction of the decision is reversed — and the decision is the thing the rules care about.

The practical test: with negotiation, anyone can get either representation. You can pull the Markdown. Google can pull the Markdown. A journalist checking whether you're lying to bots can pull the Markdown. With sniffing, the HTML version is unreachable for anyone the regex matched. That's the asymmetry that makes cloaking cloaking.

Accept is also a first-class part of HTTP, not a workaround. It has been in the spec since the beginning, and Vary exists precisely so caches handle it.

If almost nothing sends the header, haven't I lost the token savings?

For now, partly. Yes. That's the real cost and it's worth naming.

Two things make it a good trade anyway. First, the fallback is not a failure — HTML is fine. Mueller's underlying point is correct: LLMs have parsed normal web pages since the beginning. You lose efficiency, not comprehension.

Second, adoption is moving fast. Cloudflare shipped Markdown for Agents in February 2026 — Accept-header only, no user-agent detection anywhere in the docs, Vary set to include Accept. When the largest network on the web makes Accept: text/markdown the way to get Markdown from millions of origins, agents learn to send it. Build for the header and you're early. Build for the regex and you're maintaining it in 2029.

What about llms.txt or .md URLs?

Both create a second surface that drifts from the first — the exact failure Canel described. Google has compared llms.txt to the keywords meta tag: a file you maintain that consumers ignore. A separate .md URL is worse, because now you have two canonical addresses for one document and no good answer for which one to link.

One URL. Two representations. The client picks.

Doing this on your own site

1. Render Markdown from the same source as your HTML. If they come from different pipelines, they will disagree, and you'll find out from a user.
2. Negotiate on Accept, not User-Agent. Match text/markdown anywhere in the list, with optional q-params — clients rarely put it first.
3. Set Vary: Accept. Skip this and your CDN will serve Markdown to a browser at some point. It's not fun.
4. Keep one URL. No .md twin, no /md/ prefix in production, nothing extra in the sitemap.
5. Delete the crawler allowlist. All of it. It is a liability that looks like a feature.
6. Test the fallback. curl -A 'ClaudeBot/1.0' yoursite.com should return HTML. If it doesn't, you're still cloaking.

Where this goes

As discussed above, the core engineering principle remains the same.

Agents are just the same story on a shorter timeline. The population is splitting faster than any list can track — readers, actors, auditors, things nobody has built yet — and no regex is going to keep up with what they want.

So don't try. Serve one URL. Answer the question the client actually asked.

Sooner or later, agents will learn to ask for the type of content.

That's it. That's the whole policy.

References

1. David McSweeney: How [Cloudflare’s "Markdown for Agents"] Unintentionally Breaks the Web’s Trust Model — Queryburst
2. Google’s Mueller Calls Markdown-For-Bots Idea ‘A Stupid Idea’ — SEJ