A product page can pass every check in your SEO dashboard and still be invisible to the AI systems your customers are starting to shop through. Google crawls it fine. The schema validates. Rankings look healthy. But open the raw response an AI crawler actually receives, and the price is missing, the size options are gone, and there’s no sign of whether the item is even in stock.
That gap is exactly what our own research into this problem turned up, on client ecommerce sites and in the wider technical SEO discussion happening around AI crawlers right now: a genuine, repeatable way to check what’s actually happening on your own store, using free tools most technical SEOs already have open.
This guide walks through that method. Not “AI crawlers might not render JavaScript” as a vague caution, but the specific commands to run, the specific fields to check, and the specific mistake (testing from your own office network) that makes the whole audit lie to you if you skip it.
Key Takeaways
- Google renders JavaScript before indexing a page. Most major AI crawlers, including GPTBot, ClaudeBot and PerplexityBot, fetch raw HTML and do not execute JavaScript at all, confirmed independently by Vercel’s December 2024 crawler study.
- If price, availability or variant data is injected client-side rather than present in the initial HTML response, a non-rendering AI crawler never sees it, even though Googlebot and a human visitor see a normal page.
- The real audit compares three things per product URL: the raw HTML (view-source), the rendered DOM (Inspect Element), and the actual response each bot’s user agent receives via a direct request.
- Testing a bot’s user agent from your own machine or office network can return a false pass, because the CDN edge may already be serving you a cached response warmed for your IP address. Test from outside your own network for an accurate result.
- A 200 status code isn’t the finish line. The real question is whether the response contains enough product data, price, availability, and variant information, for the crawler to reconstruct the product correctly, not just whether it received a page.
- Cloudflare and similar edge/WAF setups can serve different responses to different bots by default, so one crawler getting a clean 200 doesn’t mean another isn’t getting a 403 or a challenge page, invisibly, with nothing showing up in a normal SEO dashboard.
Why This Gap Is Easy to Miss
A product page can look completely normal in Chrome, crawl fine for Google, and validate cleanly in a schema checker. But looking at the actual machine-readable response can tell a different story: price injected with JavaScript, variant information not clearly exposed, availability missing from the raw HTML, and some AI crawlers receiving different responses than others entirely. The CDN can be serving stale content in some cases too. From an SEO dashboard, everything looks fine. From a machine’s perspective, the product is incomplete.
The real question isn’t just whether a page exists, is crawlable, and validates. It’s whether the page is actually understandable, retrievable, and complete enough to get recommended by an AI system. Those are separate hurdles, not one.
Treating every AI crawler as a single audience is the wrong framing. Testing the raw response for each verified bot and recording the status code, cache age, robots decision, and which product fields are actually present turns a vague “AI can’t see it” complaint into a reproducible, bot-specific delivery check. A 200 response can still be a failure if the version of the page a crawler receives is missing the fields it needs; the real distinction is delivery success versus semantic completeness, not just whether the request succeeded.
The methodology below lays out a genuinely step-by-step process using nothing but view-source, Chrome DevTools, and curl.
Why This Is Happening: Google Renders JavaScript, Most AI Crawlers Don’t
Googlebot has run a full rendering pipeline for years. It fetches a page, executes the JavaScript, and indexes the page roughly as a browser would display it. That’s why a price injected client-side has been invisible to nobody paying attention to Google rankings specifically, Googlebot sees it fine.
AI crawlers mostly never built that pipeline. Vercel ran a study with MERJ, published in December 2024, analyzing real crawler traffic across its network, and found that none of the major AI crawlers it measured render JavaScript. That list includes OpenAI’s GPTBot and OAI-SearchBot, Anthropic’s ClaudeBot, and PerplexityBot. These bots do request JavaScript files, Vercel’s data showed a meaningful share of ChatGPT’s and Claude’s traffic hitting .js files directly, but they don’t execute what’s inside them. Content that only exists after client-side JavaScript runs is, for these crawlers, content that doesn’t exist.
That reframes the audit question. It’s not really “can this crawler reach my page.” Almost every product page returns a 200 status to almost every bot. It’s “what’s actually present in the response body before any JavaScript runs,” because that’s the only version of the page most AI crawlers will ever see.
The Audit: Comparing Raw HTML, Rendered DOM, and Per-Bot Response
This is the audit methodology, organized into steps you can run against your own product pages in order. None of it requires a paid tool.
Step 1: Check Whether Price Is in the Raw HTML
Open a product page in Chrome and press Ctrl+U (or Cmd+Option+U on Mac) to view the raw page source, then search for the price. Compare that against what Inspect Element shows once the page has fully loaded. If the price appears in Inspect Element but not in view-source, it’s being added client-side after the initial response, which means a non-rendering crawler never receives it. You can confirm the same thing visually by disabling JavaScript in Chrome DevTools (Settings, Debugger, Disable JavaScript) and reloading the page: if the price disappears, that’s the same gap an AI crawler experiences.
Step 2: Check What’s Actually in Your JSON-LD
Still inside the raw HTML from Step 1, find your product’s JSON-LD block and check it directly rather than assuming your platform’s default schema is complete. For variants: if variant data only lives in a JavaScript state object or a dropdown menu, with no hasVariant property, no separate offers per variant, and no distinct variant URLs in the markup itself, a non-rendering crawler only ever sees the default product, not its options. For availability: search the raw HTML and the JSON-LD specifically for an availability value, ideally a real schema.org URL like https://schema.org/InStock, not a plain word. If it’s missing from the initial response, most AI crawlers won’t have it either.
Step 3: Request the Page as Each Bot and Compare
This is the step that actually tests per-crawler delivery instead of assuming it. Run a direct request using each bot’s real user agent string and compare status codes, response sizes, and the body content itself:
If one bot comes back empty where another returns the price, that’s the delivery gap made concrete. Bot protection is the usual cause of different responses between bots, a challenge page or a 403 served to some user agents and not others. One important caveat worth being explicit about: spoofing a user agent string from your own laptop will not trigger IP-based bot-detection rules the way a real crawler request does, so for a final confirmation your server or CDN access logs, filtered by the real user agent strings, are the actual source of truth for what the real bots received, not just your own curl test.
Step 4: Check the Response Headers for Stale Cache
Look at the response headers from Step 3, specifically age, cache-control, and a CDN-specific header like cf-cache-status or x-cache. If a bot’s request returns a cache HIT with a high age value while the live price has actually changed, that bot is being served a stale copy of the page. This gets worse when cache varies by user agent and the bot-specific cached variant rarely gets purged, so the stale response can persist for the AI crawler long after a human visitor sees the updated price.
Step 5: Test From Outside Your Own Network
This is the step most people skip, and it’s worth avoiding that mistake deliberately. Running a bot-agent test from your own machine against your own site often teaches you nothing useful, because the CDN edge answers the request out of a cache it has already warmed for that specific IP address. The actual gaps between what a bot receives and what a human browser receives only become visible once the same test is run from outside the site’s own network. Practically, that means running your curl tests from a server, cloud instance, or VPN exit point that isn’t on your office or home connection, not from the same machine or IP you use to browse the live site every day.
Step 6: Build a Per-Crawler Audit Table, Not a Single Pass/Fail
The strongest structural fix is to stop treating every AI crawler as one audience and instead log a consistent set of fields per bot, per URL. That turns a vague complaint into a reproducible table you can actually act on:
| Field to log | What it tells you |
|---|---|
| Status code | Whether the bot was blocked, challenged, or served the page at all |
| Response size | A much smaller body than the rendered page usually means missing content |
| Cache age / cache status | Whether this bot is being served a stale, outdated copy |
| Canonical / robots directive | Whether indexing signals match what you intend for this bot |
| Product name present | Baseline check that the core content rendered at all |
| Price present | The field most often injected client-side and missing for non-rendering bots |
| Availability present | Whether the bot can tell the item is actually in stock |
| Variant data / JSON-LD present | Whether size, color, and other options are visible, not just the default variant |
Run this across your real catalog rather than one sample page by scripting the curl step over a list of product URLs, then flagging any row where price, availability, or variant data comes back missing for a given bot. For a larger store, this is the difference between an afternoon spot-check and an actual audit you can hand to a developer with specific URLs attached.
Real Current User-Agent Strings for the Major AI Crawlers
Verified directly against each company’s own current documentation, not secondary roundups. Treat exact version numbers as likely to change; identity verification by published IP range is the more durable method where a company offers one.
| Crawler | robots.txt token | Renders JS? | Notes |
|---|---|---|---|
| GPTBot (OpenAI) | GPTBot | No | UA includes GPTBot plus a version number OpenAI says is subject to change; confirmed via OpenAI’s own bot documentation |
| ClaudeBot (Anthropic) | ClaudeBot | No | Anthropic’s own support docs verify identity by published IP range rather than a single fixed UA string; it collects content for model training and honors robots.txt |
| PerplexityBot | PerplexityBot | Not rendering, per the Vercel study’s broader finding | Used for Perplexity’s own search index, not foundation-model training per Perplexity’s own bot docs; separate from Perplexity-User |
| Perplexity-User | Perplexity-User | Live fetch, not a background crawl | Fires when a real user asks a question in Perplexity; per Perplexity’s own docs, it generally ignores robots.txt since a person, not a background crawler, triggered it |
| Google-Extended | Google-Extended | N/A, not a crawler | Confirmed via Google’s own crawler documentation: it has no separate HTTP user agent of its own. It’s a robots.txt control token applied to content ordinary Googlebot already fetched, governing Gemini model training and grounding use only. Google states it has no effect on Search ranking or inclusion. |
| Googlebot | Googlebot | Yes, full rendering | Included as the baseline comparison; this is the one crawler the “looked fine in my SEO dashboard” check was actually testing against |
The Edge Layer Makes This Worse: Different Bots Can Get Different Answers
Most of what looks like an AI-crawler-specific problem isn’t the site doing anything on purpose, it’s the edge. Cloudflare, which a large share of ecommerce sites sit behind, announced in July 2025 that it would default to blocking known AI crawlers on new domains unless a site owner explicitly allowed access or opted into a pay-per-crawl arrangement. Cloudflare has since refined that in its bot changelog into separate Search, Agent, and Training crawler categories, with new defaults (effective for new domains from September 15, 2026) blocking Training and Agent crawlers on ad-supported pages while leaving Search crawlers allowed. The practical effect for a merchant: one AI crawler can get a clean response while another gets a 403 or a bot-protection challenge page, entirely at the edge, with nothing about it showing up in a standard SEO crawl report. This is exactly why Step 3’s per-bot request test matters more than a single “is my site crawlable” check.
What This Means for Shopify and Other Ecommerce Platforms
Most modern ecommerce themes, Shopify’s included, render a meaningful amount of product-page content client-side: live inventory counts, variant-specific pricing, and sometimes the base price itself, especially on apps that layer dynamic pricing or personalization on top of the page. None of that is a problem for Google. It can be a real problem for an AI shopping assistant trying to answer is this in stock or how much does the medium cost, if the answer only exists after JavaScript the assistant never runs. Running the audit above against your own product templates, not just a sample page but the actual variant and availability logic your theme uses, is the only way to know which side of that gap your store is on.
This is also the kind of audit our team runs directly for ecommerce clients. If you’d rather have someone run this per-bot comparison against your real product catalog than do it page by page yourself, it sits under South Asia Digital’s Shopify SEO services and our broader technical SEO services, not as a hard sell here, just worth knowing it’s a service line we actually run, not just a topic we write about.
For the parts of this checklist Shopify already automates, canonical tags, sitemap generation, base schema, see our separate Shopify SEO checklist. And if llms.txt came up in your own research as a possible fix for this gap, it’s worth reading why it isn’t one: see our piece on what llms.txt actually does (and doesn’t) before relying on it for this specific problem.
Frequently Asked Questions
Do AI crawlers like GPTBot and ClaudeBot render JavaScript?
No, based on Vercel’s December 2024 study of real crawler traffic across its network. GPTBot, ClaudeBot, PerplexityBot, and most other major AI crawlers fetch raw HTML and do not execute JavaScript, unlike Googlebot, which runs a full rendering pipeline before indexing a page.
If my product page ranks fine in Google, does that mean AI crawlers can see it too?
Not necessarily. Googlebot renders JavaScript before evaluating a page, so content injected client-side, like price or variant data, can be fully visible to Google while being completely absent from the raw HTML most AI crawlers receive. A page can pass every normal SEO check and still hand an AI crawler an incomplete product.
What is the simplest way to check if my price is JavaScript-injected?
Open the product page in Chrome, press Ctrl+U to view the raw page source, and search for the price. If it doesn’t appear there but does show up in Inspect Element after the page loads, it’s being added client-side and a non-rendering crawler won’t see it. Disabling JavaScript in Chrome DevTools and reloading the page shows the same gap visually.
Why does testing a bot’s user agent from my own computer give unreliable results?
Two separate reasons. First, your CDN’s edge may already be serving your own IP address a cached response, so the test reflects your cache, not a real crawler’s experience; testing from outside your own network avoids this. Second, spoofing a bot’s user agent string from your own machine does not trigger the IP-based verification that bot protection systems use on real crawler traffic, so your test can pass or fail differently than what the actual bot receives. Server or CDN access logs, filtered by the real bot’s verified requests, are the more reliable source.
Does a 200 status code mean an AI crawler received my full product data?
No. A 200 response only confirms the request succeeded, not that the response body contains the product name, price, availability, and variant data a crawler needs to represent the product correctly. Check the actual response content, not just the status code.
Does Cloudflare block AI crawlers from my site automatically?
It can. Cloudflare began defaulting to blocking known AI crawlers on new domains in July 2025, and has since split that into Search, Agent, and Training crawler categories with its own default rules. Depending on your site’s Cloudflare configuration, one AI crawler can receive a normal response while another gets blocked or challenged, invisibly, with nothing showing up in a standard crawl report.
Is Google-Extended the crawler that reads my site for Google AI Overviews?
No. Per Google’s own documentation, Google-Extended has no separate crawler or user agent of its own. It’s a robots.txt control token applied to content that ordinary Googlebot has already fetched, and it only governs whether that content can be used to train or ground Gemini models. Google states it does not affect a site’s inclusion or ranking in Search.
What should I check in my product schema specifically for AI crawlers?
Check that your JSON-LD, as it exists in the raw HTML before any JavaScript runs, includes a real availability value such as https://schema.org/InStock, and that variants are represented with hasVariant, separate offers, or distinct variant URLs rather than only existing inside a client-side dropdown or state object. If this data only appears after rendering, a non-rendering AI crawler never sees it.
How do I audit this across my whole product catalog instead of one page at a time?
Script the per-bot curl request over a list of your product URLs, logging status code, response size, cache age, and whether price, availability, and variant data are present in each response. Flag any row where a field is missing for a given bot, which turns a page-by-page manual check into a repeatable audit you can hand to a developer with specific URLs attached.
Will adding an llms.txt file fix this?
No. llms.txt is a proposed convention for pointing an AI system toward a curated summary of your site; it doesn’t change what a crawler receives when it requests your actual product page. If price, availability, or variants aren’t in the raw HTML response, an llms.txt file doesn’t put them there.
Is this a new SEO problem, or the same JavaScript-rendering issue technical SEO has dealt with for years?
Mostly the same underlying issue with a new audience. Server-side-rendering-versus-client-side-rendering concerns are not new, but many sites quietly stopped worrying about them once Google proved it could render JavaScript reliably. AI crawlers mostly never built that same rendering pipeline, so the old advice applies again, this time for an audience that can’t be assumed to catch up the way Google did.
