Websites are turning away AI agents that browse and shop for people, but most of the new friction is not showing up in robots.txt files. Across 3,658 top-ranked websites tracked by CrawlIndex, the number blocking at least one AI answer crawler slipped from 646 to 643 between Nov. 1 and Nov. 15, according to its November report. The barriers agents hit instead tend to come from site terms and bot defenses, set company by company.
The gap drew attention on Sunday after signulll, an X account with more than 270,000 followers, wrote that "the sheer number of websites now blocking ai agents is astonishing." The poster said they often take screenshots of videos to share with their agents because the underlying sites refuse access.
What the crawler data shows
CrawlIndex, run by Fidget Labs BV in the Netherlands, checks robots.txt files and homepages for up to 3,692 Tranco-ranked domains each night. Its November report shows 17.6% of measured sites block at least one answer-surface crawler and 4.5% block all of them.
Training crawlers are blocked far more often than fetchers that act on a user's request. OpenAI's GPTBot is blocked by 14.6% of sites, while its ChatGPT-User agent is blocked by 8.4%, the data shows. User-triggered agents are the part that is creeping up: Anthropic's Claude-User gained eight blocking sites this month and Perplexity-User gained seven.
Instagram was one of the sites that changed course. On Nov. 14 it began blocking user agents including ChatGPT-User and Claude-User, CrawlIndex data shows.
Across the wider web the share is smaller. Piperic, which indexes 71,281,670 live domains, reports that 5.0% block at least one AI crawler and only 0.64% block ChatGPT-User.
Where agents actually get stuck
Retailers and airlines are setting their own rules. The Verge reported on Nov. 10 that Meta's Muse agent could no longer pull eBay listings for a reader. An eBay spokesperson later said there was no blanket ban, and that eBay allows some agent-assisted browsing under its policies while blocking "unauthorized automated scraping" when detected.
The same report said Amazon blocks external agents from browsing or shopping, while Shopify has opened its catalog of about a billion listings to shopping agents. United Airlines updated its terms on Nov. 11 to require written permission before any AI agent searches or books on its site, Search Engine Journal reported. AgenticFeed covered the United change on Nov. 11.
Some blocks are accidental. Walmart has said its human-verification checks were not meant to stop Muse, and Zillow has acknowledged that its anti-scraping defenses sometimes catch agents it otherwise allows, Engadget reported.
Network providers also shape the defaults. Since Sept. 15, new domains on Cloudflare block AI training bots and agents on pages that show ads unless the owner changes the setting, according to Engadget. CrawlIndex found blocking rates of 25.1% among sites behind Fastly against 2.6% behind Google Cloud.
Why companies say no
Security is one stated reason. JPMorgan Chase CFO Jeremy Barnum said on the bank's October earnings call that agents using shared logins and screen scraping to move money are "not secure," Engadget reported.
Commercial control is another. United CEO Scott Kirby told an event in Chicago, "I don't want to create another middleman," according to The Wall Street Journal as cited by Search Engine Journal.
Demand through agents is still small. Uber said it fulfilled only four rides through third-party agents in the third quarter, and DoorDash said it handled "several hundred" agent orders against 993 million total orders last quarter, The Verge reported.
A few sites are charging instead of refusing. Nine domains answered an agent with an HTTP 402 Payment Required response, CrawlIndex data shows, which the group describes as access being metered and sold rather than refused.
Source: CrawlIndex (Fidget Labs BV), “The state of AI crawler access, November 2026”
Additional sources: Piperic, “The State of AI on the Open Web, November 2026”; The Verge, “eBay is restricting third-party AI agents”; Engadget, “Why some websites are blocking AI agents from accessing the web”; Search Engine Journal, “United lets search engines index its site, makes AI agents ask first”; signulll on X, “Post on websites blocking AI agents”
