Free research report · November 2026 edition
Every scan run through the free Arobis AI Visibility Checker tests whether ChatGPT, Gemini, Claude and Perplexity can actually read a website, and how well it is set up to be cited. This report aggregates all 2,997 scans of 2,934 websites from 1 June to 31 October 2026, with the full dataset free to download and cite.
The Website AI Readiness Report is a free, monthly benchmark of how ready real websites are for AI search. It is built by Arobis AI, an AI Search Demand Generation agency for B2B SaaS, from the scans people run through our free AI Visibility Checker. No survey, no estimates: every number comes from a live, automated scan of a real website.
2,997 scans of 2,934 websites, aggregated into one dataset. Each scan tests whether the crawlers behind ChatGPT, Gemini, Claude and Perplexity can read the site, and how well its structured data, headings, content and llms.txt set it up to be cited.
Buyers now ask AI assistants for shortlists, and the engines answer from what they can crawl and understand. This report shows where websites actually fail that test today, which signals separate the sites that pass, and how fast adoption is moving month by month.
Compare your own checker score with the tables below. If you write about AI search, quote any statistic with a link back: the data is CC BY 4.0 and every finding is written as a single citable sentence. If you want the gaps closed for you, that is the work Arobis AI does for B2B SaaS.
Jump to a section
Each statistic below is a single, citable sentence. All figures come from the 2,997 scans in this edition unless a source is named.
Who is in this sample. These are not the top 1,000 websites. They are 2,934 websites whose owners or marketers chose to check their AI visibility, which makes this the most AI-aware slice of the web we know of. Treat the adoption numbers as a ceiling for the wider web, not an average. See the methodology.
The AI Readiness Score is a 0 to 100 composite of six things an AI engine needs before it can read, understand and cite a website: crawler access, structured data, semantic structure, content and entity signals, llms.txt, and technical health. The full weighting is in the methodology. Across 2,997 scans the mean is 79.7 and the median is 85.
74.6% of websites clear the 70-point AI-ready line, which is the threshold for the green Verified AI-Ready badge. The distribution is top-heavy because the sample is self-selected: people who run an AI visibility check tend to have already invested in their site. Even so, 25.4% of these AI-aware websites still fall short, and 4.6% score below 40.
Most AI crawler studies read robots.txt and stop there. We do that too, for seven crawler tokens, but we also fetch every homepage live with each engine's real crawler user agent and check whether a usable page comes back. The two measurements disagree by a wide margin.
91.7% of websites publish a robots.txt file. Explicit AI crawler blocking is rare in this sample: GPTBot is disallowed on 3.8% of sites, and only 4.4% of sites disallow any of the seven AI tokens we check. That is far below the 19 to 25% GPTBot block rates reported for the top 1,000 to 5,000 websites by Presenc AI and HasData, which fits: this sample is made of businesses that want to be found by AI, not publishers protecting content.
Blocked means the robots.txt group for that token disallows the site root. 17 sites (0.6%) could not be evaluated because robots.txt was unreachable.
For the 2,917 scans with live probe data, we requested the homepage over HTTPS with each engine's crawler user agent. Full means a 200 response with a real page body. HTTP only means HTTPS was refused but plain HTTP served the page. None means the crawler got a block page, an error, an empty body, or nothing within seven seconds.
ChatGPT's crawler is turned away by 18.6% of websites and Claude's by 16.0%, while Googlebot, which feeds Gemini and AI Overviews, gets a full page from 93.5%. The gap is not a robots.txt decision. It is bot-protection defaults in CDNs and firewalls that ship with "AI scraper" rules switched on, which treat GPTBot and ClaudeBot as threats and Googlebot as a guest.
17.2% of websites that explicitly permit OpenAI's crawlers in robots.txt still return nothing usable to GPTBot. Their owners believe they are open to ChatGPT. They are not. This is the single most common AI visibility problem we find, and the owner almost never knows about it because every human browser test passes.
Pattern worth naming: 6.3% of websites serve full pages to Googlebot and PerplexityBot but block both GPTBot and ClaudeBot. That exact signature is the fingerprint of a managed bot-protection rule set, not a deliberate policy. Sites where GPTBot got no page average 61.9 points; sites that served it a full page average 84.2.
llms.txt is a plain-text file at the site root that hands AI systems a curated map of what a site is about. No major engine has formally committed to reading it, yet adoption among sites that care about AI visibility is climbing fast. 50.2% of the websites in this report publish one, and the rate among newly scanned sites has risen every month.
The gap is the point. In the general web, llms.txt is a single-digit phenomenon. Among businesses actively working on AI visibility it is approaching a coin flip, and platform defaults are part of the story: several website builders and e-commerce platforms now generate the file automatically. If you want a free one, use the llms.txt generator.
Does it correlate with readiness? Sites with llms.txt average 91.2 points; sites without average 68.2. Be careful with that number: llms.txt is worth exactly 10 points in our model, so 10 of the 23.0-point gap is mechanical. The other 13.0 points are real: teams that bother with llms.txt have usually also fixed their schema, headings and crawler access.
JSON-LD is how a website tells machines what it is in unambiguous terms. 78.9% of homepages include at least one JSON-LD block, which sounds healthy until you look at what is inside. Only 65.0% declare an Organization (or a subtype such as LocalBusiness or ProfessionalService), which is the one entity that lets an AI engine connect a page to a named company. FAQPage, the type most closely tied to answer-style citations, appears on 18.9%.
A homepage can declare several types, so shares do not sum to 100%. WebSite and WebPage are frequently emitted by CMS plugins without any human input, which is why they lead the table.
Structured data carries 20% of the composite score, so part of each gap is built in. The size of the gaps is still telling: the 21.1% of websites with no JSON-LD at all average 50.7 points, deep in the "needs work" band, and they rarely fail on schema alone.
AI engines extract answers from readable text and clean heading structure. The table ranks ten homepage signals by how common they are. HTTPS is effectively universal. Everything an engine uses to understand who a site is and what it says is far patchier.
18.7% of homepages have no H1, usually because the headline is an image, a styled div, or rendered by JavaScript after load. An AI crawler that does not execute scripts sees a page with no title statement at all.
The median homepage carries 966 words. The 7.9% with fewer than 100 words are mostly JavaScript-rendered single-page apps and image-led landing pages: they look fine to a human and are close to empty for a crawler. The median server response time in the sample is 605 ms, so speed is rarely the problem. Text that exists in the HTML is.
The same signals matter differently to each engine, so every scan also produces four engine-specific scores. The ranking is stable across the whole sample: Gemini and Perplexity level at the top, then Claude and ChatGPT about eight points behind.
The spread is almost entirely a crawler access story. Gemini leans on Googlebot, which nearly every site welcomes. ChatGPT and Claude depend on GPTBot and ClaudeBot, which 18.6% and 16.0% of websites turn away. For a B2B company, that means the engine buyers use most for vendor research is the one most likely to be unable to read the vendor's own site.
Each row is the cohort of websites scanned in that month, so this is a trend in who shows up and what they have already done, not a re-scan of the same sites. Two things move in the right direction: llms.txt adoption climbs 15.0 points and explicit GPTBot blocking in robots.txt falls from 8.3% to 4.0%. One thing does not: the share of sites that turn GPTBot away on a live fetch stays high, 22.0% in June and still 18.4% in October. Owners are opening the door in robots.txt while their infrastructure keeps it shut.
Live probe data starts on 1 June 2026 and covers 2,917 of the 2,997 scans; the last column uses that subset (173 scans in June).
No single fix makes a site AI-ready, but the signals stack. Sites that have done both of the two most visible pieces of work, publishing llms.txt and shipping JSON-LD, average 93.7 points. Sites with neither average 48.6, a 45.1-point gap that is far larger than the 30 points those two signals control directly.
Read in order of impact, the data says: first make sure GPTBot and ClaudeBot get a real page (check your CDN and firewall, not just robots.txt); second, declare who you are with Organization schema and one clear H1; third, put real text in the HTML; and only then add llms.txt and FAQPage on top. That is also the order our free checker lists fixes in, and it is the first month of every Arobis AI engagement.
The same order runs through our AI SEO services: our technical team implements the schema, entity pages, llms.txt and internal links on your site, and you approve each change before it goes live.
Source. Every figure comes from scans run through the free Arobis AI Visibility Checker between 1 June and 31 October 2026: 2,997 completed scans of 2,934 distinct websites. A handful of sites were scanned more than once; repeat scans are counted as separate observations. Scans that failed to reach the site are excluded.
What a scan does. The checker fetches the homepage, robots.txt and /llms.txt, parses the HTML for headings, metadata, Open Graph tags, JSON-LD and readable word count, and reads robots.txt for seven AI crawler tokens: GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-Web, Google-Extended and PerplexityBot. It then fetches the homepage live, once per engine, using the real crawler user agent for GPTBot, Googlebot, ClaudeBot and PerplexityBot (Google-Extended is a robots.txt token only, so Gemini is probed with Googlebot). A probe counts as a full page when the server returns 200 with at least 5 KB of body over HTTPS; a challenge page large enough to pass that floor would be counted as full, so the silent-block figures are conservative.
The score. Six dimensions, each 0 to 1, weighted into a 0 to 100 composite: crawler access 25%, structured data 20%, semantic structure 20%, content and entity signals 15%, llms.txt 10%, technical health 10%. A robots.txt disallow is decisive for an engine; otherwise the live probe decides (full = 1, HTTP only = 0.5, none = 0). Bands: 0 to 39 needs work, 40 to 69 getting there, 70 to 89 strong, 90 to 100 excellent. Per-engine scores reuse the same dimensions with engine-specific weights, listed in the engine table above. The scoring is deterministic and identical for every site.
Bias, stated plainly. This is a self-selected sample of websites whose owners or marketers wanted to know their AI visibility. It over-represents marketing-aware small and mid-sized businesses, agencies and SaaS companies, and under-represents publishers, enterprises and the long tail. Adoption figures here are therefore an upper bound for the web at large. Correlations between a signal and the score are partly mechanical, because the signal contributes to the score; we say how much wherever it matters. 25.8% of sites were submitted with a www prefix.
Cadence. This page is refreshed on the first working week of every month with the new cohort added and the headline numbers, tables and dataset re-generated from the same queries. The edition and last-updated date appear at the top and in the page's structured data.
The aggregate dataset behind every table on this page is published as JSON under a Creative Commons Attribution 4.0 licence. Use it in articles, decks and research; link back to this page as the source. Individual scanned domains are not included.
Arobis AI (November 2026). Website AI Readiness Report 2026: What Real AI Visibility Scans Reveal. https://arobis.ai/ai-readiness-report
According to Arobis AI's analysis of 2,997 website scans, 18.6% of websites return no usable page to ChatGPT's crawler while only 3.8% block it in robots.txt.
Journalists and researchers: for a custom cut of the data (by industry, platform or country) email ran@arobis.ai.
An AI readiness score measures how well a website is set up to be read, understood and cited by AI engines such as ChatGPT, Gemini, Claude and Perplexity. The Arobis AI version is a 0 to 100 composite of six dimensions: crawler access, structured data, semantic structure, content and entity signals, llms.txt and technical health. Across 2,997 scans in 2026 the average is 79.7.
It depends on how you measure. In robots.txt, only 3.8% of the websites in this report disallow GPTBot. On a live fetch with GPTBot's real user agent, 18.6% of websites return no usable page. Most ChatGPT blocking is done silently by CDN and firewall bot rules, not by robots.txt.
50.2% of the 2,934 websites in this report publish an llms.txt file, rising from 37.9% of sites scanned in June 2026 to 52.9% in October. Studies of the top 1,000 and top 10,000 websites report 8.7% and 5.6%, so adoption is far higher among businesses actively working on AI visibility than on the web at large.
Gemini. 93.5% of websites served a full page to Googlebot, the crawler that feeds Gemini and Google AI Overviews. Perplexity followed at 92.7%, Claude at 83.4% and ChatGPT at 80.8%.
No AI engine has formally committed to reading llms.txt, so treat it as low-cost insurance rather than a ranking factor. In this dataset sites with llms.txt score 91.2 on average against 68.2 without, but 10 of those points are the file's own weight in the model and the rest reflects that llms.txt adopters usually also have proper schema, headings and crawler access.
Every number comes from scans run through the free Arobis AI Visibility Checker between 1 June and 31 October 2026. The aggregate dataset is published as JSON under a Creative Commons Attribution 4.0 licence, so you can quote it, chart it and republish it with a link to this page. Individual domains are not shared.
Run the free Arobis AI Visibility Checker at arobis.ai/ai-visibility-checker. It performs the same scan used in this report, returns your score and the four engine scores in about 25 seconds, with a fix plan available through a short form, and sites scoring 70 or higher can embed a Verified AI-Ready badge.
The free checker runs the exact scan behind this report: live crawler probes for four AI engines, schema, headings and llms.txt, with a fix plan available through a short form. Score 70 or higher and you can show the Verified AI-Ready badge on your site.