# Why Most Websites Will Never Be Recommended by AI

- **URL:** https://brianwinum.com/why-most-websites-will-never-be-recommended-by-ai/
- **Published:** 2026-09-16

At a glance Key Takeaways A large share of AI invisibility is self-inflicted through robots.txt. Academic research shows AI-blocking by reputable sites rose from 23% to nearly 60% between September 2023 and May 2025, while misinformation sites block almost nothing, meaning the quality web is donating its presence in AI answers to lower-quality sources. Blocking the wrong crawlers costs citations and traffic. Domains blocking GPTBot show a median ChatGPT citation propensity of 0.003 versus 0.417 for domains that allow it, and Rutgers/Wharton research found blockers experienced a 23.1% decline in monthly visits. Training bots and retrieval bots are separate policy decisions. 71% of top news publishers block retrieval bots while trying to block training, cutting themselves out of citations unnecessarily, when allowing OAI-SearchBot, ChatGPT-User, and Claude-SearchBot preserves citation eligibility regardless of your training-data stance. Citation-worthy structure is now quantified. SE Ranking's 129,000-domain study found pages with 120 to 180 words between headings earn the most ChatGPT citations, FAQ schema slightly underperforms no FAQ schema, and referring domains remain the strongest single predictor. Generic content cannot survive a concentrated citation layer. With retrieval weighting shifting toward community and reference sources, the only durable strategy is information gain: original data, proprietary methodology, and claims only you can make. A few months back I audited a site that had everything money can buy. Custom design that probably won somebody an award, sub-second load times, content produced by a real editorial team. The owner wanted to know why competitors kept showing up in ChatGPT answers while his brand never did. So I ran the test I now run before opening a single crawl report: I asked ChatGPT, Claude, Gemini, and Perplexity the questions his customers actually ask, in a handful of phrasings, and recorded who got named. His competitors appeared over and over. His brand appeared zero times, across every assistant, every phrasing. And before anyone thinks I'm throwing stones from a glass house: the first time I ran this test on one of my own properties, the results were humbling enough that I didn't share them with anyone for a week. That silence is what sent me down the rabbit hole this article climbs back out of. Here's what I've learned from running this audit dozens of times since: most websites will never be recommended by AI, and the reasons sort into five failure modes that show up with depressing consistency. The most common one is the one almost nobody suspects. Sites aren't just being overlooked. A staggering number of them blocked the AI systems themselves. The Thesis, Stated Plainly If AI can't read your site, it can't recommend you. When someone asks an AI assistant a question, the system expands it into synthetic background queries, retrieves passage-level chunks from sources it can access and disambiguate, and synthesizes an answer from what it pulled. Every step of that pipeline is a filter. Fail at access, and nothing downstream matters. Fail at entity resolution, and your passages never enter the candidate pool. Fail at citability, and you enter the pool and lose anyway. If AI can't read your site, it can't recommend you. The five blockers below follow that pipeline in order. Fix them in order, too. Blocker 1: You Blocked Them Yourself This is the failure mode I find most often, and the data on its scale genuinely surprised me when I dug in. A large-scale academic study of robots.txt directives tracked AI-blocking across the web's most visited domains and found that blocking by reputable sites rose from 23% in September 2023 to nearly 60% by May 2025, with reputable sites disallowing an average of 15.5 different AI user agents. The blocking wave is real, it's recent, and it's concentrated among exactly the sites you'd want representing your industry. Now the finding that should keep every serious marketer up at night: the same research found that misinformation sites block fewer than one AI agent on average. Read that again. The quality web is systematically opting out of AI training and retrieval while low-quality sources leave the door wide open. The corpus these systems learn from and ground against is being adversely selected in real time, and every reputable site that blanket-blocks is donating its seat at the table to whoever stayed. Your absence is someone else's presence. What does blocking actually cost the blocker? Two data points, both recent. Research from academics at Rutgers Business School and The Wharton School found that publishers blocking AI crawlers via robots.txt experienced a 23.1% decline in total monthly visits and a 13.9% decline in human-only browsing. And cloro's July 2026 analysis of 1,058 prominent domains cross-referenced robots.txt rules against actual citation behavior and found that domains blocking GPTBot showed a median ChatGPT citation propensity of 0.003 citations per Google-organic appearance, versus 0.417 for domains that allow it. That's not a modest penalty. That's the difference between existing and not existing in the answer layer. The honest caveat, which cloro's researchers make themselves: this is correlational data, and domains that block crawlers may differ in other ways. But the pattern is engine-specific in exactly the way a real access effect would be. Blocking OpenAI's crawlers tracks with vanishing ChatGPT citations. Blocking PerplexityBot collapses Perplexity citations while barely denting ChatGPT. Spurious correlations don't usually respect corporate org charts. So why are smart publishers doing this to themselves? Mostly because they're conflating two different things. BuzzStream's study of top news publishers found that 79% block AI training bots, which is a defensible intellectual property position. But 71% also block retrieval bots, the crawlers that fetch content when an AI system needs to ground a live answer or produce a citation. Those publishers set out to say "don't train on my content" and accidentally said, "don't cite me either." The distinction matters because the major AI companies split these functions into separate user agents. OpenAI runs GPTBot for training, OAI-SearchBot for its search index, and ChatGPT-User for live fetches. Anthropic mirrors the structure with ClaudeBot and Claude-SearchBot. Google separates Google-Extended (AI training) from Googlebot entirely. That separation is a policy instrument, and cloro's data shows sophisticated publishers already using it: training crawlers get blocked at two to five times the rate of live-retrieval bots across their dataset. You can protect your training-data position and keep your citation eligibility. It's two lines of robots.txt, not a philosophical crisis. Your diagnostic, and I mean today: open yoursite.com/robots.txt and read it. Based on my experience, a meaningful share of the blocks I find were never a decision anyone made. A security plugin shipped them, a CDN toggled a default, a developer pasted a "block all AI" snippet from a forum in 2023, and nobody has looked since. The quality web is opting out of AI while the junk web stays in. Your absence is someone else's presence. Blocker 2: Your Content Renders for Googlebot but Not for AI I went deep on this in the pillar article, so here's the operative summary plus what the audits keep teaching me. Vercel and MERJ's analysis of AI crawler behavior established that the major AI crawlers fetch JavaScript files but do not execute them. No rendering. The lone exception is Gemini, which rides Googlebot's rendering infrastructure. So a client-side rendered site can hold position one in Google, because Googlebot renders JavaScript, while reading as an empty shell to the crawlers behind ChatGPT, Claude, and Perplexity. The audit pattern is predictable once you know to look. Headless and SPA builds fail whole-site. Otherwise healthy sites fail partially: pricing tables populated client-side, content hidden in tabs and accordions that only mount on interaction, schema injected by JavaScript after load, reviews pulled in from a third-party widget. The page a human sees and the page GPTBot receives are two different documents, and only one of them contains the reasons anyone would recommend you. Two tests, ninety seconds total. View source (the actual source, not the DevTools elements panel, which shows the rendered result) and search for a sentence from your main content. Then curl your key pages with an AI crawler user agent string and read what comes back. If the answer to "is my value proposition in the raw HTML" is no, this blocker is yours. Blocker 3: Machines Can't Figure Out Who You Are Access gets you crawled. It doesn't get you understood. The next filter is entity resolution, and I want to repeat the framing from the pillar article because it changes how you prioritize: in retrieval systems, entity ambiguity doesn't rank you lower; it filters you out of the candidate pool. A system that can't confidently determine who you are will not risk naming you, and a system that confuses you with a similarly named business somewhere else may do something worse than ignoring you. The failure inventory from my audits: no Organization schema or a bare-bones version with a name and a logo. No Person schema for authors, so every article floats authorless in machine terms. No sameAs corroboration pointing to the profiles that confirm your identity. And naming inconsistency, the quiet killer, where the site says one brand name, the LinkedIn page a variant, the directories a third, and the machine sees three weak fragments instead of one strong entity. On top of entity signals sits the emerging declaration layer: llms.txt and its variants, plus newer proposals like Google's Open Knowledge Format, files that describe your site to AI systems directly. I've covered these standards at length, and I'll hold myself to the same honesty standard here that I ask of everyone selling AI visibility services: llms.txt is a proposed standard, no major crawler operator has formally committed to consuming it, and SE Ranking's citation research found no measurable effect of llms.txt on ChatGPT citations in current data. Anyone telling you it's a ranking factor today is ahead of the evidence. So why do I still implement it, and why did I build LLMS Amplifier to generate and maintain these files automatically? Because the cost is near zero, the agentic web these files anticipate is visibly being built, and the failure mode of a stale declaration file is worse than none at all, which makes automation the sane path. It's infrastructure for where this is going, not a lever for where we are. I'd rather tell you that plainly than sell you a miracle. Blocker 4: Nothing on the Site Is Worth Citing Now the uncomfortable one, because it has no technical fix. Your site can be crawlable, renderable, and perfectly disambiguated, and still lose every citation because the content gives the machine no reason to pull it. Remember the unit of competition: passages, not pages. And the citation research keeps converging on what makes a passage win. SE Ranking's study of 129,000 domains across 216,524 pages found structural and authority patterns that should reshape how your team writes. Pages with section lengths of 120 to 180 words between headings earned the most citations, which is a near-perfect description of a self-contained retrievable chunk. Referring domains emerged as the strongest single predictor, with a threshold effect around 32,000 referring domains where average citations nearly doubled. Presence on multiple review platforms corresponded to several times the citation rate of absent domains. And my favorite finding, because it punctures a piece of conventional wisdom: pages with FAQ schema averaged fewer citations than pages without it, 3.6 versus 4.2. The industry spent two years bolting FAQ markup onto everything as an "AI optimization," and the data suggests the models care whether your information is well organized, not whether it's technically annotated. Structure is for the reader; the markup is icing. The other side of citability is volatility, and it's worth staring at. Citation pattern research covered by PPC Land documented a period in which ChatGPT's referral traffic dropped by half as OpenAI reweighted its retrieval system, shifting citations toward Reddit and Wikipedia until three domains controlled 22% of all ChatGPT citations. The lesson isn't despair; it's differentiation: when the reference layer is dominated by community consensus and encyclopedic sources, generic content that restates the consensus has nothing to offer. The only durable citation strategy is saying something the corpus doesn't already contain. Original data, proprietary methodology, named expertise, a claim only you can make. I've been calling this Catalyst Content for years, and the retrieval era has turned it from a content strategy into a survival requirement. Blocker 5: No Reason to Trust You The final filter assumes everything above passed. The machine can read you, resolve you, and your content says something worth repeating. The remaining question is whether you're a source it can afford to name. The trust deficit shows up in audits as absence: author bios that are two sentences and a stock photo, an about page written like a brochure, no editorial transparency, third-party profiles that contradict each other on basic facts. Individually, these look cosmetic. Collectively, they deprive the system of corroboration, and corroboration is how machines approximate trust. It's telling that in the citation research above, the strongest signals (referring domains, review platform presence, brand mentions across independent sources) are all forms of other people vouching for you in ways you don't fully control. The blockers above get you read. This one gets you believed. It's also the deepest topic in this series, so I'm giving it two full articles: one on why digital trust and provenance are starting to matter more than backlinks, and one on building authority with evidence instead of links. Consider this section the trailhead. The Audit Sequence I Actually Use Everything above compresses into an afternoon of diagnosis. Here's the sequence, in pipeline order. First, the mention audit. Ask ChatGPT, Claude, Gemini, and Perplexity your customers' questions in several phrasings. Record three things: who gets named, what gets said about you, and what gets said that's wrong. Inaccuracy is its own emergency and jumps the queue. Second, robots.txt review against the training-versus-retrieval split. Every disallow line should be a decision someone actually made, and the retrieval and user-fetch agents should almost certainly be allowed. Third, server log pull. Filter for GPTBot, OAI-SearchBot, ChatGPT-User, ClaudeBot, Claude-SearchBot, and PerplexityBot. You're checking status codes and coverage: are they getting 200s on your content-rich pages, or bouncing off redirects and empty shells? Fourth, raw HTML verification on your money pages. View source, find your core claims, curl with an AI user agent if you want certainty. Fifth, entity consistency pass. Organization and Person schema present and complete, sameAs pointing to profiles that exist, one canonical brand name everywhere. Sixth, the citability review, which is the hard one: open your five most important pages and ask what claim on each page only you could make. If the honest answer is none, you've found next quarter's real project. Some findings are afternoon fixes. Robots.txt edits and schema gaps close in days. Rendering architecture and citability are roadmap items. But you can't sequence what you haven't diagnosed. Invisibility Is a Choice Now Here's the reframe I want you to leave with. Two years ago, being absent from AI answers was mostly bad luck: the systems were opaque, and the levers were unknown. That excuse has expired. The blocking data shows a huge share of invisibility is self-inflicted, the rendering data shows exactly which sites machines can't read, and the citation research is now specific enough to tell you how long your sections should be. The failure modes are known, measurable, and mostly fixable. Which means the sites that stay invisible from here are, in a real sense, choosing it. The systematic version of the audit above, every check, in order, with the scoring rubric I use on client work, is coming in my AI-Friendly Website Checklist article next in this series. In the meantime, run the mention audit this week. If you find something surprising in what the machines say about you, or don't say, I'd genuinely love to hear about it. The practitioners comparing notes right now are the ones writing the next playbook. Frequently asked questions Questions, answered Direct answers on AI crawler blocking, robots.txt strategy, and why sites that rank in Google stay invisible in AI answers. Does blocking GPTBot stop ChatGPT from citing me? The data says blocking correlates with near-zero citation propensity (0.003 versus 0.417 for allowers), and OpenAI's training and search crawlers are separate, so the safer policy is deciding on GPTBot independently while allowing OAI-SearchBot and ChatGPT-User. Why does my site rank in Google but never appear in AI answers? The most common causes are robots.txt rules blocking AI crawlers, JavaScript-rendered content that AI crawlers can't execute, weak entity signals, and content that restates what higher-authority sources already say. Which AI crawlers should I allow in robots.txt? Allow the retrieval and user-fetch agents (OAI-SearchBot, ChatGPT-User, Claude-SearchBot, PerplexityBot) to stay citable, and make a separate, deliberate decision about training crawlers like GPTBot, ClaudeBot, and Google-Extended. How do I check what AI models say about my business? Ask ChatGPT, Claude, Gemini, and Perplexity your customers' actual questions in several phrasings, then record who gets named, what's said about you, and what's said that's wrong.