Make an existing website ready for AI agents

Two kinds of agent visit

• Agents that read: search crawlers that index pages for AI answers, fetchers that retrieve a page because a user asked about it, and coding agents reading your documentation.
• Agents that act: an assistant inside your page, a browser agent, or a user's own AI client calling tools your site exposes.

Readable content serves the first group. Usable, well-described actions with clear permissions serve the second.

Let search and user-triggered crawlers in

Google says its AI Overviews and AI Mode need no special optimization: a page must be indexed and eligible for a snippet, and normal SEO fundamentals apply, including allowing crawling in robots.txt and in your CDN or hosting infrastructure. See Google's guide to AI features.

AI providers publish separate user agents for search indexing, user-triggered fetches and model training, so you can choose each independently:

Blocking a search user agent can remove your pages from that provider's search answers. OpenAI and Perplexity state that user-triggered fetches may not follow robots.txt, so robots.txt is not a complete access control. If your firewall or bot protection challenges unknown clients, check its rules against the IP ranges each provider publishes. One possible policy:

# Appear in AI search and answer user requests, but opt out of training.
User-agent: GPTBot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: *
Allow: /

Sitemap: https://www.example.com/sitemap.xml

Publish content agents can read

• Put primary content in the HTML response. Text that appears only after client-side JavaScript runs may be missed by fetchers that do not execute it.
• Use real headings, lists and tables, and link text that says where the link goes.
• List canonical pages in a sitemap and give each page a canonical URL, a specific title and a description.
• Keep structured data truthful. Google asks that structured data match the visible text on the page; never add reviews, ratings or dates the page does not show.
• Offer plain-text entry points for technical readers. An llms.txt file is a community proposal that lists your key pages for language models. Treat it as a convenience for agents that look for it, not as a ranking signal. For a protocol, a Markdown page an agent can follow end to end is more useful: the Bunkhouse agent trial protocol is one example.

Make actions usable by agents

An agent acting in your page reads the same structure assistive technology does. Accessible names matter: a button labelled "Save invoice" is easier to use correctly than an unlabelled icon. Label form fields and use real buttons and links.

Some operations are not visible on the page at all, such as server-only routes or workflows the user has not opened. Describe them explicitly. Bunkhouse reads a JSON hints file served from your own origin, where each operation declares whether it reads or writes:

{
  "version": 2,
  "privateSelectors": ["#login-form"],
  "operations": [
    {
      "id": "inventory.list",
      "name": "List inventory items",
      "description": "Read one page of inventory items.",
      "effect": "read",
      "api": { "kind": "api", "origin": "https://app.example.com", "method": "GET", "pathTemplate": "/api/items" },
      "inputSchema": { "type": "object", "properties": {} }
    }
  ]
}

WebMCP is a draft Community Group Report from the W3C Web Machine Learning Community Group. It lets a page declare tools for agents through document.modelContext. Treat it as a draft that may change. The Bunkhouse assistant discovers WebMCP tools your page declares: tools annotated readOnlyHint: true are treated as reads, and other tools are shown to the user for review before they run.

Keep users and your backend in control

• Authorize on the server. An agent working through your interface and API should face the same checks as the user it acts for.
• Review writes. In Bunkhouse, API writes, consequential WebMCP tools, clicks and submits are shown to the user before they run.
• Mark private content. Bunkhouse never reads password, payment or file fields, and skips any element marked data-agent-private.
• Keep credentials out of the model. Authorization headers stay in the page and are attached only to requests to allowed origins.

The Bunkhouse security guide covers keys, origins and approvals.

Let users bring their own agent

With external agents turned on, a signed-in user can connect their own MCP client to your site through an MCP server URL of the form https://api.bunkhouse.ai/mcp/<public key>. Bunkhouse acts as the OAuth authorization server, so your host needs no OAuth server, new CSP domain or identity change. Calls run in the user's open, signed-in tab with the same skills and approval dialogs as the Bunkhouse pane, and the host's credentials never leave that tab. The same setting publishes those skills to in-page agents as WebMCP tools.

Calls only run while the connected tab is open, and actions that drive page controls need the tab visible.

Checklist

1. robots.txt and your CDN allow the search and user-triggered crawlers you want.
2. Primary content is in the HTML response, with clear headings and descriptive links.
3. Every canonical page is in your sitemap with a specific title and description.
4. Structured data matches what the page shows.
5. Controls have accessible names, and operations agents cannot see are described.
6. Writes need user review, private regions are marked, and authorization stays on your server.

Try an agent on your own page

A coding agent such as Claude Code, Codex or Cursor can run a free, disposable Bunkhouse agent trial on a dummy page of yours, with no account, card or model API key. To compare chat assistants, in-app agents and custom builds first, read how to add AI to your website.