Can Claude Scrape a Website? Our 12-Page Test

Yes, Claude can scrape a website, as long as the page opens for a plain request. Of 12 product pages, it read 7 correctly on its own, returned 1 in another currency and was blocked on 4.

Can Claude scrape a website? The short answer

What separated Claude’s answers was the store’s response to its request: every page that opened gave a price, and every refusal ended the attempt before there was anything to read.

A small Shopify store served the assistant its Polish version, so one answer was 204 PLN instead of the 47.95 EUR we saw in a browser: the right page, in another currency. The four failures were Darty, Etsy, Back Market and Hermès, which all returned HTTP 403.

Five of the 12 sites protect themselves against a plain request: those four, plus that same Shopify store, which sent HTTP 429 to a simple request. We took the 12 pages from the list of example pages in the ScrapingBot repository and chose them to mix open and protected stores. The 5 protected sites are therefore our choice, not a measure of how many stores block requests, and that list may also favor the API.

How we tested Claude on 12 product pages

We ran the test on October 22, 2026. The 12 pages come from 12 different shops: E.Leclerc, Boulanger, Darty, Etsy, Back Market, Hermès, and six stores on Shopify, WooCommerce, PrestaShop and Magento: Fashion Nova and five smaller shops. We describe the five smaller shops by platform only. The same measurements feed our ecommerce web scraping guide, which goes deeper on why product pages block requests.

The exact prompt we used

We sent one call per page and no follow-up message; inside that call, the assistant could make several attempts. We ran Claude Code in non-interactive mode on the Sonnet model and allowed two tools in the command: reading a web page and searching the web. Tools already allowed in our local settings also let it run terminal commands. This is the prompt; a placeholder stands in for the URL:

Get the product on this page: name, price (number), currency, in stock (true/false). Answer only with a JSON object {"name":..,"price":..,"currency":..,"in_stock":..,"source_url":..,"note":..}. source_url is the exact page where you read the price. If you cannot access the page or a value is not on it, use null and explain in note. URL: <url>

We ask for the exact page where the price was read, so a price taken from somewhere else would show. We also say what to do when a value is missing, which gives the model a clean way to say it could not read the page.

How we checked each price

On the day of the test, we opened each page in a browser from France and noted the displayed price. Each answer then went into one of four classes, fixed before we read the answers: right (within 2% of the displayed price, same currency), right page but another currency or country version (a geo variant), failure reported as a failure, or a wrong value given as correct (a silent error).

Which tools Claude had in our setup

When the page-reading tool returned a 403, the session also ran a direct curl request on Darty, Back Market and Hermès, and got the same 403. On Etsy, it tried a browser tool that failed to connect. None of these extra attempts changed a result.

What Claude got right, and what blocked it

Open stores: real prices in about 20 seconds

On the 7 open stores, Claude took between 11.5 and 28.7 seconds per page, and 19.5 seconds at the median. Each of its 7 prices matched the displayed price: 26.95 EUR on Fashion Nova’s Shopify store, 14.90 EUR on a WooCommerce store in Greece, 183.17 EUR on a French PrestaShop store, 49.67 EUR on a Magento spirits store, 13.31 EUR on E.Leclerc, 449.99 EUR on Boulanger and 304.00 EUR on a PrestaShop store in Spain. Three of these stores sit behind Cloudflare and served the page anyway.

The notes attached to the answers show what the values rest on. On E.Leclerc, the note says that the fetch tool had returned a summary made by a small model, not the raw HTML, and that the “in stock” label next to the color option might refer to one color or one seller only.

Boulanger is the page where the two runs disagree: the assistant read 449.99 EUR there, while the ScrapingBot API returned an error for the same URL, as the API results below show.

The Spanish PrestaShop store carries two prices for the same item: 304.00 EUR on the page and 301.49 EUR in its Product markup. One pre-tax price of 251.24 EUR explains both: the Spanish VAT rate of 21% on the page, 20% in the markup. The assistant returned 304.00 EUR, the price a shopper sees, and the gap only shows if you read the markup next to the page.

Protected stores: four 403s and no guessed price

On Darty, Etsy, Back Market and Hermès, the answer was the same four times: HTTP 403 and no price. Darty’s note reads: “I did not read any price or stock status, and I did not guess one from memory or from another site.” It adds that you would need a real browser session or an authorized product feed or API.

The Hermès note says the server answered 403 to two different requests, that no product data was visible, and that name, price, currency and availability are null. It guessed nothing from the URL. The Etsy note says the same for the fetch tool and adds that the browser tool failed to connect.

The fifth protected store, the small Shopify brand that answers a plain script with HTTP 429, let Claude through. That is where the Polish page came from.

No invented price in 12 answers

A common worry about assistants is that they fill gaps with plausible numbers. We did not see it in our 12 answers: each of the 8 prices in them had been read on the page, 7 in euros and 1 in Polish zloty, and each of the 4 failures came back as null and gave a reason.

What the documentation says about Claude’s limits

Anthropic’s web fetch tool does not render JavaScript

The documentation for the Anthropic API’s web fetch tool (read on October 22, 2026) states that it “does not support websites dynamically rendered with JavaScript”. That page describes the API tool. Our test used Claude Code, which has its own page-reading tool, so we do not assume that page describes the tool we used. The warning still holds: if a price appears only after a script runs, a plain fetch will not see it.

The same page lists the errors the tool can return, including one for HTTP errors, the kind four of our five protected stores sent back, and one for URLs blocked by restrictions such as robots.txt.

What blocked us was anti-bot protection

None of our 4 failures was a rendering problem. They were HTTP 403 responses, the code a server sends when it refuses a request. The assistant reported the 403 from its fetch tool on all four sites and from a direct curl request on three of them.

Cloudflare’s bot documentation (read on October 22, 2026) describes this from the site owner’s side: AI bots are classed by behavior (Search, Agent, Training), and each customer sets their own preferences. A refusal from Back Market or Hermès, both behind Cloudflare, is the site’s own rule, applied automatically. Darty sits behind DataDome and Akamai, and Etsy behind DataDome, two other protections that set their own rules.

The same 12 pages with an API collecting first

Results with the API in front

For the second run, we called the ScrapingBot Retail API on the same 12 pages, then made one call to the model, without tools, to put the 12 JSON responses into a table. That analysis call took 13 seconds and kept the empty fields of the two failed pages as null.

The API run gave 8 right, 2 geo variants and 2 failures:

• The two geo variants were Fashion Nova, which returned 23.98 USD instead of 26.95 EUR, and Etsy, which returned 32.17 USD instead of 34.43 EUR. Both sites adapt the price to the country of the request.
• Boulanger failed: the API reported “The website returned HTTP 400”.
• The PrestaShop store in Spain came back with empty fields.

On the 5 protected sites, the API returned a page for all of them. Four were retried through a premium proxy, announced in the response at 10 credits each. ScrapingBot also forced a premium proxy on Hermès, without saying so in the response; we know it from a rule in the repository, not from the calls themselves.

Collection took a median of 3.1 seconds per page over the 12 pages, from 1.1 to 39.0 seconds, and 2.6 seconds on the 7 open stores. The retried pages were the slowest: 14.1, 17.8, 21.8 and 39.0 seconds. The 12 calls used 65 credits in total; the published rates account for about 57 of them, and we could not attribute the rest.

The two runs cover different pages. In the API-first run, 10 pages returned a price. The assistant alone read Boulanger and the Spanish store, where the API failed, and the API read Darty, Etsy, Back Market and Hermès, where the direct reads were blocked. Together they cover all 12 pages.

Darty, Etsy, Hermès and Boulanger each have a scraper page on this site: the Darty scraper, the Etsy scraper, the Hermès scraper and the Boulanger scraper.

Reproduce it: connect ScrapingBot to Claude with MCP

We did not use the MCP server in the test: the API calls and the analysis ran as separate steps. The server lets the model call the API itself, without moving JSON around by hand.

The Model Context Protocol is, in its own words, “an open-source standard for connecting AI applications to external systems” (official introduction, read on October 22, 2026). The ScrapingBot MCP server uses it to give Claude Code and Claude Desktop access to the API as tools.

You need your ScrapingBot username and API key, plus Node.js 18+ and Git. For Claude Code, this is the command from the MCP server page:

claude mcp add scraping-bot \
  -e SCRAPING_BOT_USERNAME=your_username \
  -e SCRAPING_BOT_API_KEY=your_api_key \
  -- npx -y github:sostag/scraping-bot-mcp

For Claude Desktop, the same server goes into the claude_desktop_config.json file. The MCP server page has the exact block.

The server exposes 8 tools:

• scrape_raw_html: the raw HTML of any public page, JavaScript rendering optional
• scrape_retail: a product page, returned as structured data
• scrape_real_estate: a property listing
• scrape_booking: hotels and rentals on Booking.com, Airbnb, Expedia, Hotels.com and Abritel
• scrape_grocery: a grocery product with price and nutrition values
• scrape_search_engine: Google or Bing results as JSON
• scrape_social_media: public profiles, posts and searches
• get_social_media_result: the result of a social media job that ran longer than a minute

Credit costs follow the API: Raw HTML, Retail, Real Estate and Booking calls start at 1 credit, JavaScript rendering takes a call to 5, a premium proxy to 10 and both together to 25, and search engine and social media calls cost 30. Failed requests are free, and plan prices are on the pricing page.

Once the server is connected, you ask in plain language, for example: “Scrape these three product URLs with scrape_retail and put title, price, currency and stock in a table.” The assistant then calls scrape_retail on each URL and builds the table from the responses.

What the API returns that Claude can check

The Retail response carries more than a price: title, description, image, price, shippingFees, currency, isInStock, EAN13, ASIN, ISBN, brand, category, siteURL, statusCode, captchaFound, htmlLength, premiumProxyUsed and a notice field. The last few are what make the second step checkable. When the Spanish store’s response returned empty fields, statusCode said 200 but htmlLength was 0 and isHtmlPage was false: no page had come back, so the failure happened during collection.

captchaFound is the field to ignore: it was true on 5 responses that carried the right price, so it is no reason to reject an answer. The useful checks are statusCode, null fields, error, premiumProxyUsed and notice. The API documentation lists the parameters and response fields.

What about ChatGPT?

ChatGPT was not part of the test, and the only material we have on it is what OpenAI publishes. Its page on bots and user agents (read on October 22, 2026) separates the crawlers: OAI-SearchBot is for search visibility, GPTBot for training, and ChatGPT-User for actions a user triggers inside ChatGPT. About that last one, the page says: “Because these actions are initiated by a user, robots.txt rules may not apply.”

That sentence describes how OpenAI classifies the traffic. It tells us nothing about what ChatGPT would have read on our 12 pages.

FAQ

Can ChatGPT scrape a website?

We cannot say from our data, because ChatGPT was not part of the test. The section above quotes what OpenAI publishes about the user agent behind those requests.

Can Claude write scraping code for me?

Yes, and Claude Code can also run the code it writes. We did not measure how often that code works. On our 4 blocked pages, better code would not have helped: the 403 came before any parsing, and code that reads a page needs the page first.

Is it legal to scrape a website with AI?

That depends on the site’s terms, the data you collect, personal data rules and your country, and doing it through an AI assistant changes none of these. The legal section of our ecommerce web scraping guide goes through them.