Skip to content
Learn

AI agents and AI crawlers

In this lesson

Section titled “In this lesson”
  • Explain how agentic search differs from an AI answer.
  • Describe the three ways an agent reads a web page, and make a page easier for agents to use.
  • Explain what MCP is and why structured access can be better than scraping.
  • Tell the training, search and user bots of OpenAI, Anthropic, Google and Perplexity apart.
  • Choose a policy for AI crawlers with robots.txt and with network controls such as Cloudflare pay per crawl.

This lesson has two halves. The first half is about agents: AI systems that do tasks on websites for a person. The second half is about AI crawlers: the bots that collect your pages. Both halves answer one question: what can an AI system do with your site, and what can you control?

Section titled “Agentic search”

On day 07, an AI system found passages and wrote an answer. Agentic search goes one step further. The system does not only answer. It takes actions for the user: it browses several pages, compares options, and can start a task such as a booking or a purchase.

Google’s AI optimization guide (July 2026) defines AI agents as “autonomous systems that can perform tasks on behalf of people, such as booking a reservation or comparing product specifications”.

A real example came in August 2025. Google added “agentic capabilities” to AI Mode, starting with restaurant reservations. A user can ask for a table with a party size, a time, a place and a type of food. Google says that AI Mode then searches “across multiple reservation platforms and websites” and finds real-time free slots. It uses three things: live web browsing from Project Mariner, direct partner integrations (OpenTable, Resy, Tock and others), and the Knowledge Graph with Google Maps. At launch, it was a Labs experiment for Google AI Ultra subscribers in the US. The user still made the last step on the booking page.

This example shows the two routes that agents use:

  1. Browse the website like a person, page by page.
  2. Connect through a structured interface, such as a partner API or a protocol.

The rest of the first half covers each route. For most sites today, the first route matters more, because few sites have agent interfaces yet.

How agents browse

Section titled “How agents browse”

A browser agent is an agent that opens real pages in a browser and clicks, types and scrolls. Google’s web.dev guide “Build agent-friendly websites” (April 2026) says that agents “operate on a machine-readable representation of your site”. They use three views:

View What the agent reads Strength Weakness
Screenshot An image of the rendered page, read by a vision model Sees size, color and position “Slow and expensive”
HTML (the DOM) The nested elements and their attributes Shows which button belongs to which product Does not show what a styled <div> does
Accessibility tree The roles, names and states of the interactive elements A clean map of what each control does Only as good as your markup

The accessibility tree is a summary of the page that the browser builds for assistive technology, such as screen readers. Modern agents combine the views. They take the list of controls from the DOM and the accessibility tree, and check the layout in a screenshot.

So the advice for agents is mostly old advice. The web.dev guide lists these practices:

  • Use <button> and <a> for actions, not a <div> with a click handler.
  • Connect each <label> to its input with the for attribute.
  • Do not cover controls with transparent overlays (“ghost elements”).
  • Keep the layout stable, so that a button stays in the same place.
  • Make interactive elements visible, and larger than 8 square pixels.

The guide’s own summary is the key point: “Everything we suggest to make a site ‘agent-ready’ also makes sites better for humans.” You met machine-readable content and rendering on day 03b. Agents add one more reason to get them right.

How do you see agents in your logs? Some identify themselves. Google lists a Google-Agent user agent, “used by agents hosted on Google infrastructure to navigate the web and perform actions upon user request”. Cloudflare reported in August 2025 that ChatGPT Agent signs its requests with Web Bot Auth, a proposed standard where a bot proves its identity with a cryptographic signature. A user agent string alone can be faked, so a signature is a stronger proof.

MCP and structured access

Section titled “MCP and structured access”

Browsing is slow and fragile. A button can move, and a pop-up can block the agent. Structured access is the other route: the site gives the agent a defined interface, with named actions and clear data.

The main standard is the Model Context Protocol (MCP). Anthropic released it as open source on 25 November 2024. The MCP docs describe it as “an open-source standard for connecting AI applications to external systems”, and compare it to “a USB-C port for AI applications”. ChatGPT, Claude, Visual Studio Code and many other tools support it.

MCP has three roles:

  • The host is the AI application, for example Claude or an IDE.
  • The host makes one client for each connection.
  • A server is a program that gives context to clients. It can run on the same computer or on the web.

A server offers three main kinds of things: tools (actions, such as “search products”), resources (data, such as a file or a record) and prompts (templates for tasks). Messages use JSON-RPC 2.0, a simple format for requests and responses.

For a website owner, MCP matters when agents must do things with your data, not only read it. An online shop or a booking service can publish a remote MCP server. Then an agent calls check_availability directly, and does not guess from the HTML.

Two newer standards build on the same idea:

  • WebMCP is a proposed web standard from Chrome. A page defines tools in HTML forms or JavaScript, and a browser agent calls them. In February 2026 it was in an early preview program. web.dev warns that it “is in active development and may change”.
  • Universal Commerce Protocol (UCP) is a standard for agentic shopping, from discovery to checkout. Google, Shopify, Walmart, Amazon, Microsoft and others are listed as co-developers. It supports MCP. Google says that protocols like UCP “will allow Search agents to do more”.

Which matters most? For most sites, clean HTML matters most today, because every browser agent can use it. MCP and UCP matter most for shops and services that want agents to complete tasks. Google’s guide calls all of this something to explore “if you have extra time”. It is not a requirement for Search.

Managing AI crawlers

Section titled “Managing AI crawlers”

Now the second half. An AI crawler is a bot that fetches pages for an AI company. The most important idea in this half is that AI bots have three different jobs:

Job What the bot does Effect of a block
Training Collects pages that can train future models Your future content is excluded from training
Search Builds an index for AI search answers Your site does not appear in that AI search, or appears less
User request Fetches one page because a user asked The AI cannot read your page for that user

Each big company now uses separate bots for these jobs. So you do not need to choose “all AI or no AI”. You can, for example, block training and allow search.

Cloudflare’s data shows why this matters. Across its network, 80% of AI crawling in the year to July 2025 was for training, 18% for search and 2% for user actions. Cloudflare also measures the crawl-to-refer ratio: how many pages a platform crawls for each visitor that it sends back. In July 2025, the ratio was about 38,000 to 1 for Anthropic and 194 to 1 for Perplexity. Training crawls use your content but send almost no visitors.

The tools you have

Section titled “The tools you have”

robots.txt is the main tool. You met it on day 03. For AI bots, two rules from RFC 9309 are important. First, a bot follows the group that names it, and uses the User-agent: * group only when no group names it. Second, the rules “are not a form of access authorization”. A bot can ignore them.

User-triggered fetchers are a special case. Google says that its user-triggered fetchers “generally ignore robots.txt rules”, because a user asked for the page. OpenAI says that for ChatGPT-User “robots.txt rules may not apply”. Perplexity says that Perplexity-User “generally ignores robots.txt rules”. Anthropic is different: it says that a Claude-User block prevents retrieval for user queries.

Network controls work where robots.txt does not. A firewall can block requests by user agent and by IP address. OpenAI, Perplexity and Google publish the IP ranges of their bots, so you can check that a request really comes from them. Anthropic gives one warning: an IP block can stop its bots from reading your robots.txt, so it may not work as a clean opt-out.

The four named bots

Section titled “The four named bots”

The roadmap names four bots. Here they are with their related bots, from each company’s docs in October 2026.

Company Training Search User request
OpenAI GPTBot OAI-SearchBot ChatGPT-User
Anthropic ClaudeBot Claude-SearchBot Claude-User
Google Google-Extended (a token only) Googlebot Google-Agent and other user-triggered fetchers
Perplexity none listed PerplexityBot Perplexity-User

GPTBot collects content that “may be used in training” OpenAI’s foundation models. OpenAI says that each setting is “independent of the others”. A site can disallow GPTBot and allow OAI-SearchBot, and still appear in ChatGPT search. Some articles say that a GPTBot block reduces ChatGPT search visibility. OpenAI’s docs do not say this: OAI-SearchBot controls search. After a robots.txt change, OpenAI’s search systems need about 24 hours to adjust.

ClaudeBot collects content that “could potentially contribute” to Anthropic’s model training. A block “signals that the site’s future materials should be excluded” from training data. Anthropic says that its bots honor robots.txt, support the non-standard Crawl-delay rule, and do not try to bypass CAPTCHAs.

Google-Extended is not a crawler. It is a product token: a name in robots.txt that controls a use, not a bot. It has no user agent of its own. Google crawls with its normal bots, and the token decides whether that content may train future Gemini models and ground answers in Gemini Apps and on Vertex AI. Google says that it “does not impact a site’s inclusion in Google Search nor is it used as a ranking signal”. AI Overviews and AI Mode are part of Search, so a Google-Extended block does not remove you from them. To limit those features, use the Search Console opt-out from day 07 or snippet controls.

PerplexityBot builds Perplexity’s search index. Perplexity says that it “is not used to crawl content for AI foundation models”. To appear in Perplexity answers, allow it. Note: in October 2026 the roadmap.sh file for PerplexityBot showed the Google-Extended text by mistake.

robots.txt policy
Choose a robots.txt policy. Each row shows what one bot does with the file. Notice that blocking training bots leaves every search bot free, and that 'User-agent: *' also blocks Googlebot.


Trust is still an open question. In August 2025, Cloudflare reported that Perplexity used “stealth, undeclared crawlers” that changed user agents and IP addresses when blocked. Perplexity rejected this. It said that the requests came from user-driven assistants, not crawlers, and that Cloudflare could not tell the two apart. The two sides do not agree, and the facts are not confirmed by an independent source. For you, the lesson is simple: robots.txt is a request, and enforcement needs network controls.


Cloudflare controls and pay per crawl
Section titled “Cloudflare controls and pay per crawl”


Cloudflare sits in front of many websites, so it can enforce rules at the network level. Its AI Crawl Control (formerly AI Audit) is available on all plans. With it you can see which AI services visit your site, allow or block each crawler, and track which crawlers break your robots.txt rules.


On 1 July 2025, Cloudflare changed its default “to block AI crawlers unless they pay creators for their content”. On the same day it announced pay per crawl: a site sets a price, and AI crawlers pay for each page.


Pay per crawl uses an old HTTP status code: 402 Payment Required. For each crawler, the site owner picks one of three actions:


    

  • Allow: free access.
    • 

  • Charge: access at the site’s price.
    • 

  • Block: no access, with no option to pay.
    • 



A crawler must register with Cloudflare and sign its requests with Web Bot Auth. Then nobody can fake it and make it pay. The crawler can state its maximum price in a crawler-max-price header. If the price is equal or lower, it gets the page with a 200 OK and a crawler-charged header. If not, it gets a 402 with a crawler-price header. Cloudflare is the merchant of record: it handles the payments.


Site rule for this crawler
Example prices. Choose a site rule, move the crawler's maximum price, and turn the signature on and off. Notice that 'charge' acts like a block for a crawler that cannot pay, but it tells the crawler that it could pay later.


Keep the status in mind. In July 2026, Cloudflare’s docs still called pay per crawl a closed beta. It is an experiment, not yet a common way to earn money.


What matters most
Section titled “What matters most”






Priority
Action
Why






1
Decide training and search separately
They are different bots with different effects




2
Keep search bots allowed if you want AI visibility
OAI-SearchBot, Claude-SearchBot, PerplexityBot and Googlebot feed AI answers




3
Never block * by mistake
It also blocks Googlebot and removes you from Search




4
Use network controls when you need enforcement
robots.txt is only a request




5
Build clean, semantic pages
Every browser agent uses them, and people do too




6
Explore MCP, WebMCP or UCP if agents must complete tasks
Useful for shops and booking services






A news publisher may block training and charge for crawls. A shop that wants buyers from ChatGPT should allow the search bots and make its checkout easy for agents. Your choice depends on what your content earns for you.


Key terms
Section titled “Key terms”


    

  • Agentic search: AI search that takes actions for the user, such as comparing options or starting a booking.
    • 

  • AI agent: an autonomous system that does tasks on behalf of a person.
    • 

  • Browser agent: an agent that opens pages in a browser and clicks, types and scrolls.
    • 

  • Accessibility tree: the browser’s summary of the roles, names and states of the controls on a page.
    • 

  • Structured access: access through a defined interface with named actions and data, not through scraping.
    • 

  • MCP (Model Context Protocol): an open standard that connects AI applications to tools and data through servers.
    • 

  • WebMCP: a proposed web standard where a page defines tools for browser agents.
    • 

  • AI crawler: a bot that fetches pages for an AI company, for training, search or a user request.
    • 

  • User-triggered fetcher: a bot that fetches one page because a user asked; it may ignore robots.txt.
    • 

  • Product token: a robots.txt name that controls a use of content, not a separate bot, such as Google-Extended.
    • 

  • Crawl-to-refer ratio: the number of pages a platform crawls for each visitor that it sends back.
    • 

  • Web Bot Auth: a proposed standard where a bot proves its identity with a cryptographic signature.
    • 

  • Pay per crawl: a Cloudflare feature where AI crawlers pay a set price for each page, with HTTP 402.
    • 



Exercise
Section titled “Exercise”


Exercise · about 15 min
Audit the AI crawler policy of a site
You need only a browser. Pick a large news site, for example https://www.nytimes.com, and one site that you own or know.
    

  1. Open /robots.txt on each site.
    2. 

  2. For each of these bots, write “allowed” or “blocked”: GPTBot, OAI-SearchBot, ClaudeBot, Claude-SearchBot, Google-Extended, PerplexityBot. Remember: if no group names the bot, the * group applies.
    3. 

  3. For each site, write the policy in one sentence, for example “blocks training, allows AI search”.
    4. 

  4. Check one possible mistake: does any group block Googlebot, or block * with Disallow: /?
    5. 

  5. On a page of your own site, open Chrome DevTools, select a button, and open the Accessibility pane. Does the button have a role “button” and a clear name?
    6. 

  6. Write one change for your own site: a robots.txt rule or a markup fix.
    7. 

Show a possible solution
Results change over time, so yours may differ. A typical result:
    

  • A large news publisher often blocks GPTBot, ClaudeBot, Google-Extended and other training bots. Many also block the search bots, so the policy is “blocks most AI bots”. Some publishers have deals with AI companies and allow those bots.
    • 

  • A small business site often has no AI groups at all. Then every bot follows the * group, and all are allowed.
    • 

  • Step 4: a User-agent: * group with Disallow: / and no Googlebot group is a serious mistake. It removes the site from Google Search.
    • 

  • Step 5: a “button” that is a <div> often shows the role “generic” and no name. The fix is a real <button> with visible text.
    • 

  • A possible change: “Add User-agent: GPTBot, User-agent: ClaudeBot and User-agent: Google-Extended with Disallow: /, and keep the search bots allowed.”
    • 



Check yourself
Section titled “Check yourself”


A site wants to appear in ChatGPT search answers, but it does not want OpenAI to train models on its content. What should its robots.txt do?


A publisher blocks Google-Extended. What happens to its pages in AI Overviews?


A robots.txt file has one group: 'User-agent: *' with 'Disallow: /'. A user asks Perplexity to summarize one page of the site. What is the most likely result?


A browser agent cannot complete a checkout. The 'Pay' control is a styled <div> with a click handler. Why does this cause a problem?


With Cloudflare pay per crawl, a site charges $0.005 a page. A registered crawler sends crawler-max-price: $0.008. What happens?


Sources
Section titled “Sources”



Next
Section titled “Next”


Day 08 covers measurement and testing: Search Console, analytics, rank tracking and how to measure your visibility and citations in AI answers.