ChatGPT can't read your X Article

If you only publish on X, you’re writing for two search engines.

On 8 October 2026, x.com’s robots.txt lets Googlebot and Bingbot in and ends with two lines for everyone else: User-agent: * and Disallow: /. That shuts out the crawlers behind ChatGPT search, Claude and Perplexity.

Your X Article can rank on Google. It can show up in Bing and Copilot. But the AI tools a growing share of people ask instead are told not to read it.

What the file says

You can read it yourself at x.com/robots.txt. It has a shared group for Googlebot and Bingbot with a long list of allow and disallow rules, some rules for Facebook’s link previews, explicit blocks for Google-Extended and Meta’s AI crawlers, and then this:

# Every bot that might possibly read and respect this file
User-agent: *
Disallow: /

None of the AI search crawlers have a group of their own, so they fall under *. Going by each company’s own documentation, that covers:¹²³

Each company says its search crawler respects robots.txt. So none of them should be indexing X.

There are two loose ends. Each company also has a fetcher that runs when a user asks about a specific page (ChatGPT-User, Claude-User, Perplexity-User), and those don’t always follow robots.txt. But x.com is a JavaScript app: when I fetched my own profile without running JavaScript, I got the bio from the page’s meta tags and none of the posts. And whether ChatGPT also pulls in Bing results is disputed. Neither changes the point: if you want an AI to be able to read your work, X is the wrong place to keep the only copy.

Why this matters more in 2026

Search clicks are drying up. When Google shows an AI Overview, the top result gets about 58% fewer clicks, according to Ahrefs’ February 2026 study.⁴ SparkToro’s analysis of Similarweb data found 68% of US Google searches in January–April 2026 ended without a click.⁵

And the AI engines don’t agree with each other. Kevin Indig looked at 3.7 million citations and found 91% of cited URLs appeared in only one of ChatGPT, Perplexity or Google’s AI Overviews.⁶ Being visible to Google tells you nothing about being visible to ChatGPT.

What do they cite? Google’s own AI optimisation guide, published in May 2026, tells site owners to make “non-commodity” content and says you don’t need llms.txt or special markup.⁷ The May 2026 core update rewarded original sources and first-hand data, and hit thin aggregators and generic listicles.

Here’s the rule I take from all that: if an AI could write it, an AI will answer it. Tips, how-tos and “10 lessons” posts get summarised in the answer box, and nobody clicks through. Numbers you counted yourself get cited, because the AI has nowhere else to get them.

My setup

This is the setup I’m moving my X audience studies to.

1. Full version on my own site first. The whole article, with the method, the tables and the limits, as plain HTML. Numbers go in text and tables, not only in images, because AI crawlers don’t run JavaScript and can’t read a chart.
2. Indexed before it goes on X. I submit it in Search Console and through IndexNow, and wait until it shows up. X can’t carry a canonical tag back to my site, so if a word-for-word copy goes up first, Google may decide X’s copy is the original.
3. A different X Article, not a copy. Shorter, built around one graphic, with “full data and method” linking back near the top.
4. One quotable sentence near the top. The number, the sample and the date in one line, so an AI can lift it with the context attached.
5. The door open on my own site. My robots.txt allows every crawler. If your site is on Cloudflare, check its AI crawler settings too: it can block them at the edge by default on new domains.
6. A real author. A real name, a link to my X account and real published and updated dates.

I don’t bother with llms.txt. Google says Search doesn’t use it, and no major AI company says it reads it.

Check it yourself

1. Open x.com/robots.txt and scroll to the bottom.
2. Ask ChatGPT, Claude and Perplexity the question your article answers, and look at which sources they cite.
3. Check your own robots.txt and your host’s bot settings. If you’re on Cloudflare, look for the AI crawler controls in the dashboard.
4. Check your site’s referral traffic for chatgpt.com, perplexity.ai and claude.ai, and Search Console’s AI report for impressions.

If your articles only live on X, you’ll find you’re missing from two of the three.

What this can’t tell you

• Robots.txt is a request, not a wall. It shows what X asks crawlers to do, not what every crawler actually does.
• Things change. X can edit the file any day. I read it on 8 October 2026.
• The AI search market is moving. Which index ChatGPT and Claude lean on is still contested, and the click studies above are vendor datasets.