What AI Crawlers Can See on Your Shopify Store (and How to Check in 30 Seconds)

When a shopper opens your product page, their browser downloads the HTML, runs your theme's and apps' JavaScript, loads the reviews widget, converts the currency and draws the finished page. It all happens in a second, so it's easy to forget that it's several steps.

Many AI crawlers stop after the first step. They download the HTML, read the text that's in it, and move on. Anything that only appears once a script runs (often the reviews, sometimes the price, often the size chart) doesn't exist as far as they're concerned. And if your robots.txt turns them away, they never get even that far.

This guide shows you how to check both in a few minutes, what typically goes wrong on Shopify stores, and how to fix it.

The three kinds of AI crawler

"AI crawlers" aren't one thing. The big AI companies each run several, for different jobs, and robots.txt rules apply to them by name. That matters, because blocking the wrong one can quietly take you out of AI answers.

• Training crawlers (such as GPTBot, ClaudeBot, CCBot) collect public pages that may be used to train future AI models. Blocking them is a business decision about your content. It doesn't remove you from search results.
• Search and answer crawlers (such as OAI-SearchBot, Claude-SearchBot, PerplexityBot, plus Googlebot and Bingbot) build the indexes assistants search when someone asks a question. Block these and that assistant can't find or cite you.
• On-request fetchers (such as ChatGPT-User, Claude-User, Perplexity-User) open a specific page because a person asked the assistant to look at it, or because it's checking a detail while answering. Some providers say robots.txt rules may not apply to these, because a person started the request.

Two names need special care:

• Google-Extended isn't a separate crawler. It's a robots.txt token that controls whether Google may use your content for its Gemini models, including grounding answers in Gemini apps. It doesn't affect Google Search, and that includes AI Overviews and AI Mode. Those follow Googlebot.
• Applebot-Extended works the same way for Apple's AI models. Plain Applebot powers Siri and Spotlight search results.

Crawler names and rules change. Before you write rules for a specific crawler, check that provider's own documentation for its current names.

The 30-second check

Do this on one important product page first, then on a collection page and your home page.

User-agent: OAI-SearchBot
Disallow: /

curl -I -A "OAI-SearchBot" https://yourstore.com/products/your-product

What usually goes wrong on Shopify

Editing robots.txt on Shopify, safely

Shopify generates robots.txt from a theme template called robots.txt.liquid. It doesn't exist until you create it: Online Store → Themes → … → Edit code, then under Templates choose Add a new template and pick robots.

The safe pattern is to keep Shopify's default rules and add yours after them. Shopify's default template is:

{% for group in robots.default_groups %}
  {{- group.user_agent }}

  {%- for rule in group.rules -%}
    {{ rule }}
  {%- endfor -%}

  {%- if group.sitemap != blank -%}
    {{ group.sitemap }}
  {%- endif -%}
{% endfor %}

To block a single crawler, for example a training crawler you've decided to keep out, add a new group after the loop:

User-agent: GPTBot
Disallow: /

AI shopping agents: a different route in

Crawlers aren't the only way AI reads your store any more. Shopify now gives AI shopping agents a structured way to search your catalogue and check out. Open yourstore.com/llms.txt and you'll probably see Shopify's own "agent instructions" file pointing agents to it.

Agents using that route don't read your theme at all. They get your product data straight from Shopify: title, description, product type, category, tags, variants and options. So the quality of the data in your Shopify admin decides what they see.

When we looked at what this catalogue returned for a real store we use for testing (61 products sampled), a lot was missing that a shopper on the website would never notice:

• the category field came back empty for every product, and the product type was blank
• about half the products with a single variant called "Default Title"
• more than a third of products with no tags
• no rating data for any product

None of that affects how the store looks. All of it affects how well an agent can match a product to a request like "a gift under $50 for someone who likes hiking". The fix is ordinary product hygiene: set the standard Category on every product, fill in product type and tags, name your variant options properly, and write descriptions that state the facts. Our Shopify Product SEO Checklist covers each of these.

Should you block AI crawlers at all?

It depends on what you want. Here are the three common positions:

For a store, the second option is usually the most you'd want: your products stay findable, and your content isn't collected for training. Whichever you choose, make it a deliberate decision rather than the side effect of a snippet copied from a forum.

A 15-minute fix list

FAQ

If I block GPTBot, will ChatGPT stop recommending my store?

Not by itself. OpenAI uses GPTBot for training and OAI-SearchBot for ChatGPT's search features, and documents them separately. Blocking GPTBot while allowing OAI-SearchBot keeps you eligible to appear in ChatGPT's search results. Blocking OAI-SearchBot is what removes you.

Do AI crawlers run JavaScript?

Google's crawler renders JavaScript, and Bing's does to a large extent. Many AI crawlers have been observed fetching only the raw HTML. Since you can't be sure which ones render, don't rely on it: anything important should be in the HTML.

Does llms.txt control which crawlers can read my store?

No. llms.txt is a guide that points AI tools to useful content. It doesn't grant or refuse access. Access is controlled by robots.txt (and by anything sitting in front of your store, like a firewall).

Can I see which AI crawlers visit my store?

Shopify doesn't give merchants raw server logs, so you can't see every crawler request. You can see the visits AI assistants send you in your analytics: look for referrers such as chatgpt.com, perplexity.ai and copilot.microsoft.com. Our guide to checking whether AI assistants recommend your store explains how.