Ecommerce Web Scraping, Tested on 12 Stores

Ecommerce web scraping is the automated collection of product data from online stores: price, currency, stock, identifiers, sometimes reviews. We sent a plain request to 94 product pages, then tested 12 stores two ways, with Claude alone and with our ecommerce scraper API. This page gives the results, including the pages each path missed.

What ecommerce web scraping collects

A scraper turns a store page into a dated row of fields you can store and compare.

From category page to product page

Product data sits on two kinds of pages. A product page describes one item and carries the most fields. A category URL returns product listings: many products, usually with a name, a price and a link to each product page.

The usual job runs in two steps. You read the category URL to collect product URLs, then you scrape each product page for the full set of fields. Both our first check and the test used product pages only.

The fields a Retail call returns

A Retail call is one POST request with the page URL:

import requests

response = requests.post(
    "https://api.scraping-bot.io/scrape/retail",
    auth=("your_username", "your_api_key"),
    json={"url": "https://www.example.com/product-page"},
)
print(response.json())

The API documentation covers the Retail endpoint and its options. On the responses we received during the test, the Retail API returned these fields:

• product: title, description, image, brand, category
• offer: price, currency, shippingFees, isInStock
• identifiers: EAN13, ASIN, ISBN
• the fetch itself: siteURL, statusCode, captchaFound, htmlLength, premiumProxyUsed, notice

The last group describes how the page was fetched: the status the store returned, the size of the HTML received and whether a premium proxy was used. That is how you know what a call cost and whether to trust it.

Some responses also carried an images list, and the Hermès response a SKU, the store’s own item reference, in a sku field. Reviews, ratings and seller names were not among the returned fields. If you need them, request the page with the Raw HTML endpoint and extract them yourself. On our trial account, a first Raw HTML call answered with HTTP 200 in 1.5 s, for 1 credit. What you get also depends on the page: during the same trial, a Retail call on a page without Product markup returned the title and the description, with the price set to null.

What decides whether a store can be scraped

Of everything we logged, two properties of a page explained most outcomes: the format of the data inside it and the protection in front of it. Our first check measured both.

First check: a plain request on 94 product pages

On October 22, 2026, we sent one plain GET request, with a browser User-Agent, to each of 94 product pages taken from the list of example pages in the ScrapingBot repository. No JavaScript rendering, no retry: we wanted to see what a store returns to the simplest script.

• 92 pages answered with a page or a block; one more returned HTTP 400 and one did not answer.
• 37 of the 92 blocked the request (40.2%): 25 with HTTP 403, 10 with HTTP 429, 1 with a redirect to a challenge, and 1 with a challenge page sent with HTTP 200.
• Of the 55 pages served, 34 carried schema.org Product markup (62%).

The last blocked page matters to anyone who checks only the status code: it said 200 and contained no product.

In the headers, cookies and HTML of the 94 pages, we found these vendor signatures: Cloudflare on 47 pages, Akamai on 23, DataDome on 8, PerimeterX on 2, Imperva on 1, and none on 19. Some pages showed more than one.

We could identify the store platform on only about twenty pages, too few to break the results down by CMS.

Structured product data does most of the work

Many stores embed a schema.org Product block in their pages, with an Offer inside it for the price, the currency and the availability. Google’s documentation on product snippets explains that this markup lets Search show “additional product information such as ratings, review information, price, and availability”. The schema.org Offer type describes availability “for example In stock, Out of stock, Pre-order”.

For a scraper, this block is the easy path: the fields have fixed names, so a redesign that keeps the markup does not break the extraction.

The markup is not required, though: a small PrestaShop store selling computer parts in France carried no Product markup, and the API still returned the right price, 183.17 EUR.

Markup and page can also give two different prices. On a PrestaShop furniture store in Spain, the page displays 304.00 EUR while its Product markup says 301.49 EUR. Both come from one pre-tax price, 251.24 EUR: the markup applies a 20% VAT rate and the page the Spanish rate of 21%. An API that reads only the markup would inherit that gap, so compare a few markup prices with the displayed ones before you trust a new store.

Anti-bot protection decides the rest

Stores have a reason to filter automated traffic, because there is a lot of it. Imperva’s 2025 Bad Bot Report states that “automated traffic has surpassed human activity, accounting for 51% of all web traffic”. Imperva sells bot protection and does not publish its method on that page.

One way to sort that traffic is a challenge. Cloudflare’s documentation describes Challenges as a way to “verify whether a visitor to your site is a real human”, and says most visitors pass them without any interaction.

In the test below, the vendor name did not predict the outcome. Three stores showed Cloudflare and still served a plain request, while Back Market and Hermès, also behind Cloudflare, answered 403. Boulanger showed Akamai and served the page anyway.

We scraped 12 product pages two ways

How the test worked

From the same list, we took 12 product pages on 12 different stores. Six run on Shopify, WooCommerce, PrestaShop and Magento: Fashion Nova and five smaller stores. The other six are large retailers on their own stacks.

Each page went through two paths on October 22, 2026, with one call per page and per path:

• Path A, AI assistant alone. Claude (Sonnet) with its page-reading and web search tools, asked for the product name, price, currency and stock. Our settings also allowed terminal commands: on Darty, Back Market and Hermès, Claude tried the page reader, then curl, and got the same 403.
• Path B, API then AI. The ScrapingBot Retail API, one call per page, then a single Claude call without any tool that put the 12 responses into a table.

The reference was the price displayed on the page in a browser from France. We sorted every answer into four classes, fixed before reading the results:

• right: within 2% of the displayed price, in the same currency;
• geo variant: a real price in another currency, because the site adapts it to the country of the request;
• reported failure: the tool says it got no price;
• silent error: a wrong price given as correct.

Results: AI assistant alone vs scraping API

#	Store	Plain request	Browser price	A: Claude alone	B: API, then Claude
1	Fashion Nova (Shopify)	Cloudflare seen, served	26.95 EUR	26.95 EUR, right, 21.2 s	23.98 USD, geo variant, 2.6 s
2	Shopify store (home hardware, small brand)	Cloudflare, HTTP 429	47.95 EUR	204.00 PLN, geo variant, 15.1 s	47.95 EUR, right, 14.1 s, premium retry (10 credits, stated)
3	WooCommerce store (streetwear, Greece)	Cloudflare seen, served	14.90 EUR	14.90 EUR, right, 14.7 s	14.90 EUR, right, 2.6 s
4	PrestaShop store (furniture, Spain)	No protection seen, served	304.00 EUR (markup: 301.49 EUR)	304.00 EUR, right, 21.7 s	Empty fields, reported failure, 1.4 s
5	PrestaShop store (computer parts, France)	Served, no Product markup	183.17 EUR	183.17 EUR, right, 11.5 s	183.17 EUR, right, 6.6 s
6	Magento store (spirits, France)	Cloudflare seen, served	49.67 EUR	49.67 EUR, right, 15.1 s	49.67 EUR, right, 1.1 s
7	E.Leclerc (custom stack)	No protection seen, served	13.31 EUR	13.31 EUR, right, 19.5 s	13.31 EUR, right, 2.8 s
8	Boulanger (custom stack)	Akamai seen, served	449.99 EUR	449.99 EUR, right, 28.7 s	HTTP 400, reported failure, 1.5 s
9	Darty (custom stack)	DataDome + Akamai, HTTP 403	199.99 EUR	No price (403), reported failure, 18.5 s	199.99 EUR, right, 21.8 s, premium retry (10 credits, stated)
10	Etsy (custom stack)	DataDome, HTTP 403	34.43 EUR	No price (403), reported failure, 13.1 s	32.17 USD, geo variant, 17.8 s, premium retry (10 credits, stated)
11	Back Market (custom stack)	Cloudflare, HTTP 403	553.00 EUR	No price (403), reported failure, 12.3 s	553.00 EUR, right, 39.0 s, premium retry (10 credits, stated)
12	Hermès (custom stack)	Cloudflare, HTTP 403	690.00 EUR	No price (403), reported failure, 12.4 s	690.00 EUR, right, 3.4 s, premium proxy forced by ScrapingBot, not stated

	A: Claude alone	B: API, then Claude
Right	7	8
Geo variant	1	2
Reported failure	4	2
Silent error	0	0
Median time per page, 7 open stores	19.5 s	2.6 s

The silent error line reads zero on both sides. On the four sites that refused it, Claude gave the 403 as the reason, and it invented no price on any of the 12 pages.

The results split along the protection. Claude was right on all 7 stores that serve a plain request and failed on 4 of the 5 protected sites. On the fifth, the small Shopify brand, it read the page without being blocked and was served the store’s Polish version, at 204.00 PLN. The API returned a price on all 5 protected sites. Hermès went through a premium proxy that ScrapingBot forces on that site: the response says nothing about it, and we found the setting in the ScrapingBot code.

On the API side, the four premium retries were the slowest calls, at 14.1, 17.8, 21.8 and 39.0 s; no call on an open store took more than 6.6 s.

To go further on three of these retailers, start from the Darty scraper, the Etsy scraper or the Hermès scraper.

Where the API failed

Each of the two failures was visible in the response, and both are easy to catch in code.

Boulanger. The API answered “The website returned HTTP 400”. In a browser from France the page showed 449.99 EUR, and Claude alone read it in 28.7 s.

A small PrestaShop store selling furniture in Spain. The API returned HTTP 200 with empty fields and an htmlLength of 0, so no page had come back, although the page carries Product markup. Claude alone read the displayed price, 304.00 EUR.

AI assistant or scraping API: who does what

When Claude alone is enough

The 7 open stores of the test, including the two the API missed, show where an assistant fits: a small, irregular job such as checking a few competitor prices, reading a page the API misreads, or describing a store before you automate it.

It does not fit volume, because in our setup an assistant reading a page cost tens of cents in model usage, where an API call costs a fraction of a cent.

When the target blocks AI tools

The large protected retailers were where Claude stopped. Darty, Etsy, Back Market and Hermès all answered its requests with HTTP 403, and it said so. These sites did not single out Claude, since they also refused the plain request of our first check, sent with an ordinary browser User-Agent.

Getting those pages is what a scraping API is for: it fetches the page, retries through a premium proxy when the site refuses the first attempt, and returns fields you can store directly.

A workflow that keeps both

Path B is the workflow we would keep: the API collects and the assistant reads the JSON. Claude never touched a store in that path; it put the 12 responses into a table in one call of 13 s and left the API’s empty fields empty.

We would add one rule to path B: when the API returns an error or an empty price, hand that URL to the assistant. In the test, that rule recovers the two pages described above, and the two paths together cover 12 pages out of 12.

Our walkthrough on how to scrape an ecommerce website with Claude covers the setup. If you want Claude to call the API by itself, the ScrapingBot MCP server connects it to Claude Code, Claude Desktop and Cursor.

Ecommerce use cases worth the setup

Price monitoring and pricing decisions

Price monitoring means collecting competitor prices on a schedule and comparing them with yours. The first trap in the test was the currency. Two stores sent their US dollar price to the API (23.98 USD on Fashion Nova, 32.17 USD on Etsy) while our browser in France saw 26.95 EUR and 34.43 EUR.

Each is the price the store shows to a visitor from that country, so store the currency with every price and request every competitor from the same country. A chosen country costs 25 credits per request; we did not use that option in this test.

Our article on price scraping with an API covers the setup for Amazon, eBay and other marketplaces. If you want those competitor prices inside your own PrestaShop back office, the ScrapingBot PrestaShop module brings scraped data there.

Amazon, Walmart and eBay: outside our test

None of the 12 stores in our test was Amazon, Walmart or eBay. The only public benchmark we found is the Proxyway Web Scraping API Report 2025, from a comparison site that uses affiliate links: 15 protected targets, Amazon and Walmart among them, about 6,000 URLs each, sent at 2 requests per second and at 10 on targets that held up, mostly in October 2025. For each API, it reports a success rate and an average response time of successful requests, and its method and limitations are published on the same page.

On these marketplaces, where many sellers can list the same item, identifiers do the matching. The ASIN field is Amazon’s product identifier, and EAN13 lets you match the same item across retailers. ScrapingBot has dedicated pages for the Amazon scraper, the Walmart scraper and the eBay scraper.

Stock and range pages

Stock comes back in the isInStock field. Collected on a schedule, it tells you when a competitor runs out of an item, which is a signal for your own pricing.

Range pages need care: the Back Market URL in our test was a range page priced “from 553.00 EUR” in the browser, and the API returned 553 EUR with nothing in the response to say it was a starting price. That is the entry price of the range, which may not be the model you want to compare, so check that each URL points to one configuration before you compare prices.

Costs and errors to plan for

What a request really costs in credits

ScrapingBot bills in credits per request. The grid below comes from our code and documentation; failed requests are free.

Request type or option	Credits per request
Raw HTML, Retail, Real Estate, Booking	1
Search Engine, Social Media	30
With JavaScript rendering	5
With a premium proxy	10
With JavaScript rendering and a premium proxy	25
With a chosen country	25
With a chosen state	60

We checked the base rate on a new account. One Raw HTML call and one Retail call took the counter from 490 to 488, so 1 credit each.

In the test, the 12 Retail calls cost 65 credits (counter from 480 to 415). We can explain about 57 of them: four premium retries at 10 credits, Hermès at 10 under the API’s rule for its forced premium proxy, and the simple calls at 1. We could not attribute the rest.

For a budget, count the protected stores first: in the test, every premium call was on one of them.

Response checks before you trust the data

These are the errors we hit on October 22, 2026:

• HTTP 400. From the API on Boulanger, with the message “The website returned HTTP 400”. From the API again when we sent an invalid URL, this time with a message that gives an example of a valid one.
• HTTP 401 “Unauthorized”. A wrong API key, or the account email used in place of the username. The API authenticates with the username and the key.
• HTTP 200 with nothing usable. A challenge page sent with status 200 in our first check, empty fields on the Spanish PrestaShop store, and a null price on our trial page without Product markup.

So check the content as well as the status. Before a row enters your table, check statusCode, check that htmlLength is above 0, that price and currency are not null and that the currency is the one you expect for that store, and read the error message when there is one. Log premiumProxyUsed and notice next to the row, since they tell you what the call went through and what it cost.

Do not reject a row on captchaFound alone: in our test it was true on 5 responses whose price was right, Darty among them. When a check fails, send the URL to a retry or to the assistant fallback, never into the table with a blank.

Is ecommerce web scraping legal?

No single ruling settles the question.

What the hiQ v. LinkedIn ruling does and does not say

The decision we describe is hiQ Labs v. LinkedIn, handed down by the US Court of Appeals for the Ninth Circuit on April 18, 2022 (No. 17-16783). The court affirmed a preliminary injunction that barred LinkedIn from denying hiQ, a data analytics company, “access to publicly available LinkedIn member profiles”. It found that hiQ had raised a serious question: whether the Computer Fraud and Abuse Act (CFAA) and its ban on access without authorization apply to a website that requires no authorization in general but refuses access to a particular bot.

What the ruling does not say:

• It is a preliminary injunction: a provisional measure, granted under a test where serious questions on the merits can be enough. The court did not decide the case on the merits.
• It is US law. It interprets a US federal statute, the CFAA, next to a claim of interference with contract, and it decides nothing under European law.
• It does not make all scraping lawful. It concerns public profiles, one statute and one dispute.

Terms, personal data and how fast you ask

Many stores have terms of service, and some forbid automated collection, so read them for each target before you automate. Whether they bind you depends on the country and on how you accepted them.

Price, stock and identifiers describe products. Reviews and seller profiles can name people, so we follow one rule: if a field names a person, ask a lawyer before you store it.

Keep request rates low and spread out on each domain. A 429 is the site saying it receives too many requests, and in our first check, 10 pages answered a single plain request with one. On those stores, the 429 worked as a block from the very first request.

FAQ

How can I scrape data from ecommerce websites?

Collect the product URLs from category pages, send each one to a scraping API that returns fields, and check price and currency before you store a row with its date. For a few pages on stores that serve a plain request, an AI assistant such as Claude does the same job more slowly: on the 7 stores of that kind in our test, it took 19.5 s per page at the median, against 2.6 s for the API.

Can ChatGPT or Claude scrape a website?

Claude can read pages that serve a plain request. In our test of October 22, 2026, it got the right price on the 7 stores of that kind. It could not read 4 of the 5 protected stores (HTTP 403), said so, and invented no price. We did not test ChatGPT.

Is it illegal to scrape ecommerce sites?

There is no general answer: it depends on where you are, which data you take and what the site’s terms say. The hiQ v. LinkedIn ruling of April 18, 2022 is a US preliminary injunction about public profiles, not a general permission. This is not legal advice.

Can you scrape ecommerce sites for free?

For some stores, yes. In our first check, 55 of the 92 pages that answered served a plain request, and those are the pages a short script with an HTML parser can start with. With ScrapingBot, failed requests are free, and plan details are on the pricing page.

Before you write any code, paste one of your target product URLs into the live test. It uses your account credits, and when a store needs a premium retry, the response says so in a notice.