Content for Agents (Beta)

Content for Agents publishes a Markdown version of a WordPress post or page alongside its HTML version, so that AI crawlers, agents, and other tools that prefer Markdown can read a site’s content directly instead of parsing rendered HTML.

WordPress VIP worked with Pew Research Center to adapt PRC Markdown for Agents for this plugin. Pew’s cooperation and commitment to open source means VIP can bring this proven technology to all our customers.

Prerequisites

The application must run WordPress 7.0 or newer and PHP 8.2 or newer, on the WordPress VIP Platform.

Limitations

Only public, unprotected, published content gets a public Markdown version. Draft, private, and password-protected posts follow the same WordPress access checks as their HTML equivalents, and Content for Agents does not make them public.

A site’s own paywall or login rules on an HTML page do not automatically extend to that page’s Markdown. Restrictions are possible; see extending Content for Agents below.

Markdown conversion covers standard block editor content, nested blocks, and Classic Editor HTML, including quotes, citations, lists, tables, links, images, and code.

Custom blocks and embeds may not convert accurately without additional integration work. There is a fallback HTML-to-Markdown parser, but that may not capture everything in a block. If a block fails, the plugin returns nothing instead of partial content.

Enable Content for Agents

1. Add the integration to the organization. From the Integrations Center in the VIP Dashboard, an Org admin opens Content for Agents and selects + Add to Organization. See add an integration to an organization.
2. Activate it for the application. In App Integrations, activate Content for Agents for the application. For a multisite network, activate it for the specific network site.
3. Compare a few real, representative articles — including any gated articles and any articles that use custom blocks — against their Markdown output.

Access Markdown content

For a published post at /example/, these return the same Markdown:

• /example/markdown (also accepts a trailing slash)
• /example/?markdown=true
• A GET or HEAD request to /example/ with Accept: text/markdown

A site using plain permalinks advertises the query-string form instead, such as ?p=123&markdown=true; a static front page uses /?markdown=true, which leaves /markdown free for a normal page named “markdown.”

The Accept header match is case-sensitive and requires the lowercase text text/markdown as a substring of the header value, so a mixed-case or nonstandard value won’t match.

Content for Agents also adds a Markdown discovery link to supported HTML pages, so a site can rely on that link instead of constructing the URL.

The Markdown response opens with YAML frontmatter (title, description, date, author, original URL, categories, and tags, when available), then the converted content. It links back to the original HTML page, includes an estimated token count, carries the site’s Content-Signal values, and sends a noindex header so search engines are directed to the HTML page instead.

Extending Content for Agents

Standard WordPress posts and pages work with Content for Agents without any custom integration. A site needs an integration only for the specific cases it uses:

• Custom post types — add content-for-agents support with add_post_type_support() on init, after the post type is registered, or declare content-for-agents in that post type’s own register_post_type() supports array.
• Custom blocks — register a block callback on the content_for_agents_register_block_callbacks action (runs at init priority 5), so the Markdown reflects only the text, links, and child content that’s visible on the HTML page. A callback returns a Markdown string, an empty value to suppress the block, or WP_Error to fail conversion for the whole document.
• Custom summaries, bylines, or other metadata — filter content_for_agents_frontmatter to read from the same source of truth the HTML page uses.
• Paywalls or login rules — apply the same entitlement check to Markdown responses that already applies to the HTML page, using content_for_agents_can_serve_markdown, and use content_for_agents_can_cache_markdown to keep visitor-specific Markdown out of shared caches.
• Redirected or removed articles — stop serving or advertising Markdown once the HTML URL no longer represents the article.
• Data that changes outside a normal post save — call Markdown_Cache_Invalidator::invalidate_post( $post_id ) so the cached Markdown and page cache are purged. VIP sends the resulting edge-cache purges asynchronously, so the edge may take a short time to catch up.

Extension point	Contract
content_for_agents_pre_markdown	Return null for normal conversion, a string to supply the document body directly, or WP_Error to stop.
content_for_agents_after_markdown	Filter the converted body during response assembly. Direct post_to_markdown() calls don’t run it.
content_for_agents_frontmatter	Filter the metadata array; receives the post.
content_for_agents_markdown_prefix	Filter the Markdown between frontmatter and body (the title heading, by default). Return '' to omit it.
content_for_agents_can_serve_markdown	Return false to veto Markdown access for a post. Cannot override WordPress’s own access checks.
content_for_agents_can_cache_markdown	Return false to send private, no-store response headers instead of allowing shared caching.
content_for_agents_block_{block-name}	Filter a registered callback’s result for a specific block. Only runs when a callback is registered for that block.
content_for_agents_content_signal_values	Whether the site invites AI training, search indexing, and AI input use of its content, in the Markdown response headers and in robots.txt on a public site. All three default to yes. Modify these values with the filter. Return an empty array to omit the header and robots.txt entry.

Before launching an integration, compare the Markdown against the visible HTML for a standard post, each custom block type involved, an article with custom metadata, a gated article, a redirected or removed article, and an article after its underlying data changes.

Troubleshooting

A custom block fails to convert. Content for Agents returns an error for the whole document rather than a partial conversion when a block it doesn’t recognize fails to convert cleanly. Register a block callback for that block type (see extending Content for Agents, above).

A gated or private article’s Markdown doesn’t match its access rules. A redirect or login screen on the HTML page does not, by itself, protect that page’s /markdown, ?markdown=true, or Accept: text/markdown routes. Entitlement checks must be applied to Markdown responses directly.

An article’s Markdown is stale after an integration-owned data change. Content for Agents invalidates its cache automatically on standard post edits, but data an integration maintains outside the normal post-save flow needs an explicit Markdown_Cache_Invalidator::invalidate_post() call. Returning different output from a filter does not, by itself, invalidate a response that’s already cached.