2026-09-27

robots.txt, security plugins and AI shopping bots: a WooCommerce checklist

A WooCommerce store can have a perfect product catalog and still be invisible to AI shopping agents, because the crawlers never get the page. There are two places that happens: robots.txt, which asks crawlers to stay out, and firewalls (security plugins, CDNs, hosting rules), which block them outright. The first is easy to see. The second usually isn't.

This checklist covers both, with the exact user agents to look for and commands to test them.

The AI crawlers, and what each one does

Each vendor now runs separate bots for training, search indexing and user-requested fetches, and you can treat them differently.

Sources: OpenAI's crawler overview, Anthropic's crawler help page, Perplexity's crawler docs.

For shopping visibility, the search bots matter most: OAI-SearchBot, Claude-SearchBot and PerplexityBot build the indexes that product recommendations come from. The user bots matter when a shopper pastes your link. The training bots are a separate question.

A robots.txt that allows search and blocks training

If you want to be recommended but would rather not have your content used for model training, this is a reasonable starting point:

# AI search and user-requested fetches: allowed
User-agent: OAI-SearchBot
User-agent: ChatGPT-User
User-agent: Claude-SearchBot
User-agent: Claude-User
User-agent: PerplexityBot
User-agent: Perplexity-User
Allow: /

# Model training: not allowed
User-agent: GPTBot
User-agent: ClaudeBot
User-agent: Google-Extended
Disallow: /

# Everyone else: your normal rules
User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
Disallow: /cart/
Disallow: /checkout/
Disallow: /my-account/

Two details that trip people up:

• A crawler uses the most specific group that names it. A bot listed in its own group ignores the User-agent: * rules entirely, so repeat anything important (like keeping bots out of checkout) if you need to.
• Blocking training is fine; blocking search is what costs you. A blanket User-agent: * / Disallow: / or a plugin's "block all AI bots" button hides you from the search bots too.

WooCommerce URLs that must stay crawlable

When you tighten robots.txt, it's easy to block something a shopping agent needs. Keep these open:

• Product pages (/product/…/ by default) and product category pages (/product-category/…/). Category pages are how crawlers discover products you haven't linked elsewhere.
• Product images in /wp-content/uploads/. Agents show images, and feeds point to them. A rule like Disallow: /wp-content/ blocks every product photo.
• Your sitemap. WordPress core publishes one at /wp-sitemap.xml; SEO plugins often use /sitemap_index.xml. Add a Sitemap: line with its full URL at the end of robots.txt so every crawler finds it.
• The Store API (/wp-json/wc/store/v1/products) if you want headless tools and agents to read product data directly. Blocking /wp-json/ in robots.txt is common and mostly harmless for search, but a firewall rule on it is not.

Cart, checkout and account pages can stay disallowed: there's nothing there for a crawler, and keeping bots out saves your server work.

Where WordPress robots.txt comes from

WordPress serves a virtual robots.txt unless a real robots.txt file exists in your site's root folder, in which case the file wins and the virtual rules are ignored. SEO plugins usually offer an editor for the virtual one. So if your edits don't show up, check for a physical file (your host's file manager shows it).

Also check Settings → Reading → "Discourage search engines from indexing this site". With it on, WordPress asks every crawler not to index your pages.

The part robots.txt doesn't show: firewalls

Security plugins, CDNs and hosts can block crawlers by user agent, by IP reputation or by rate, and the crawler never gets far enough to read robots.txt. Common causes:

• a security plugin's "block bad bots" or "block AI crawlers" option,
• a CDN setting that blocks AI crawlers (Cloudflare, for example, has a dashboard switch for this and announced in 2025 that it would block known AI crawlers by default for new domains),
• rate limits that treat a crawler's burst of requests as an attack,
• a hosting-level firewall rule you never set yourself.

Test it

Ask for the same product page as a browser and as each bot, and compare:

URL=https://yourstore.com/product/your-product/
curl -s -o /dev/null -w "browser          %{http_code}\n" -A "Mozilla/5.0" "$URL"
for bot in OAI-SearchBot ChatGPT-User Claude-SearchBot Claude-User PerplexityBot Perplexity-User GPTBot ClaudeBot; do
  printf "%-16s " "$bot"
  curl -s -o /dev/null -w "%{http_code}\n" -A "Mozilla/5.0 (compatible; $bot/1.0)" "$URL"
done

If the browser gets 200 and a bot gets 403, 406, 429 or 503, or a challenge page, something is blocking that bot.

Read the result carefully

Some CDNs verify crawlers by IP address: a request claiming to be OAI-SearchBot from your laptop is obviously fake, so it gets blocked, while the real crawler from OpenAI's published IP ranges gets through. OpenAI publishes those ranges for each of its bots, and good CDNs use them. So a block in this test means look at your firewall's bot settings and logs, not necessarily that the real crawler is blocked. If the logs show real crawler requests (from the vendor's published IPs) being blocked, that's the one to fix.

Checking all of this from inside WordPress

The free LeyMish AI Shopping Readiness plugin does both checks from your own store:

• it reads your robots.txt and evaluates it for nine AI user agents (the ones in the table above), using the same "most specific group, longest matching rule" logic crawlers use, against a real product URL;
• it requests one of your product pages with each crawler's user agent and compares the response with a normal browser request, flagging anything blocked as a "possible block" to confirm in your firewall.

It also checks that the store is live, which catches WooCommerce's "coming soon" mode, the other common reason crawlers see nothing. Everything runs on your site with no outside calls.

The short version

1. Store live, and search engines not discouraged.
2. robots.txt allows OAI-SearchBot, Claude-SearchBot and PerplexityBot on product pages.
3. The user bots aren't blocked (they may ignore robots.txt, but a firewall still stops them).
4. No security plugin, CDN or host rule blocks those user agents. Test it, then confirm in the logs.
5. Training bots: your call. Blocking them doesn't hurt shopping visibility.