How to Scrape Dynamic Websites with Python

To scrape dynamic websites with Python, first check whether the data is already in the HTML the server sends. We fetched 45 product pages with a plain Python request on October 23, 2026: 35 already had the product price in the raw HTML. Only one page kept its price out of every response we got, and JavaScript rendering failed on it. A plain request, or a raw HTML scraping API, covers most pages at a fraction of the cost of a browser.

Check the raw HTML before you start a browser

The usual advice for dynamic content is to open the Network tab and find the request that loads the data, or to start Selenium or Playwright and let a browser run the scripts. Both work, and both cost time; a headless browser also costs money at scale. In the ScrapingBot documentation, the useChrome option is listed at 5 credits per request, against 1 for a plain one.

Before paying that, run one check. Open the product page in your browser, note the price, then fetch the same URL with Python requests and search the HTML for that number. If it is there, the page is not dynamic for your purpose, whatever framework built it.

The distinction comes from how the page is built. Addy Osmani and Jason Miller put it in two lines on web.dev: “Server-side rendering generates the full HTML for a page on the server in response to navigation”, while “Client-side rendering means rendering pages directly in the browser with JavaScript” (web.dev, Rendering on the Web, last updated January 5, 2026).

What we found on 45 product pages

The pages come from the example product URLs in the ScrapingBot repository: 55 pages that had answered a plain request the day before. On October 23, 2026, we fetched each one again from an office connection in France, with a browser User-Agent and nothing else, one request per second. 45 answered with status code 200, 6 with 403 and 4 timed out.

Our first pass found a JSON-LD product block with a price on 27 pages. For the 13 other pages that sent real content, we opened each one in Chrome, noted the displayed price when there was one, and searched for it in the raw HTML. A later check found one more JSON-LD block, on Snow+Rock, which brings the count to 28.

What the raw HTML held	Pages
The price, in a JSON-LD product block	28
The price, in the visible page text	6
The price, in an embedded JSON script	1
Unclear, see below	3
No price in any response we got	1
An anti-bot page instead of the product	5
No public price (the shop shows prices after login)	1

On the 40 pages that sent real content, a plain request was enough for 35.

Two limits apply. These are ecommerce pages picked from one repository, so read the counts as a direction rather than a share of the web. For the 28 JSON-LD pages, we checked that the block carried a price, not that it matched the displayed price.

The three unclear cases show why one fetch is not always enough:

• AS Discount, a French computer parts shop, sent our plain request a page without the price, while the API, without rendering, got 179,24 € in the HTML.
• The Equine Warehouse, a riding gear shop, carries €45.00 in a meta tag of its raw HTML, and the page shows €44.00 to a French visitor once its scripts have run.
• Uncommon Goods sent our office an empty shell with no visible text. The API’s plain call, from another location, got a page rendered by the server with the US price, $35.00.

Where the data hides in the HTML of a dynamic web page

On the pages we tested, when the price was not in the visible text, it sat in a script that the page ships for search engines or for its own JavaScript: a JSON-LD block on 28 pages, an embedded JSON state on 1.

JSON-LD product data

A <script type="application/ld+json"> block describes the product for search engines in the schema.org vocabulary. Its offers hold the price, which schema.org defines as “The offer price of a product, or of a price component when attached to PriceSpecification and its subtypes” (schema.org Offer). Nike’s Air Force 1 page in our sample is built with Next.js, a JavaScript framework, and still ships this block in the first HTML. Parse it with json.loads and you have name, price, currency and availability without touching the layout.

Watch the spelling of the type. Snow+Rock writes it as "http://schema.org/Product" rather than "Product", and our first count missed it.

Prices inside an embedded script

Some apps send their initial state as JSON in the page, so the scripts can draw it without a second request. Uniqlo’s Australian product page held the socks’ price as "prices":{"base":{"currency":{"code":"AUD","symbol":"$"},"value":14.9}, for $14.90 on screen. Search for the number in both forms, 14.90 and 14.9.

Prices in the page text

Six pages simply printed the price in the HTML, such as Auchan’s 464,50 € for a projector and Razer’s $649.99 for a gaming chair. Razer’s page also repeats the price in a script, so a scraper can read either, with a BeautifulSoup selector or a JSON parser.

Most of our empty responses were bot checks, not JavaScript

Six responses to our plain request were nearly empty: about 3 KB of markup or less, or no visible text at all. Five of them were anti-bot pages:

• Zara and Bershka sent a two-kilobyte page with a bm-verify token that reloads after five seconds, an Akamai check.
• Micromania answered “Request unsuccessful” with an Incapsula incident ID.
• Hollister sent a page titled “Client Challenge”, saying JavaScript is disabled.
• Nordstrom sent a page made of scripts only, with no text.

The sixth, Uncommon Goods, was the empty shell described above, which the same URL served complete to the API.

A bot check is the site deciding not to serve a request, and rendering does not change that decision. It is a different problem from JavaScript, and our guide to scraping without getting blocked treats it separately. Look for the markers before you conclude anything: bm-verify, _Incapsula_, “Client Challenge”, or a page with no visible text.

Python code to check a page before rendering it

This script fetches a page once and tells you where the price is: in a JSON-LD block, in the text, in an embedded script, or nowhere. It also flags the anti-bot pages we met, plus three markers we added without meeting them in this test (px-captcha, captcha-delivery, cf-chl), and pages with almost no visible text. You pass the URL and the price you see in your browser.

import json
import re
import sys

import requests

HEADERS = {"User-Agent": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/130.0 Safari/537.36"}
BOT_CHECK = re.compile(r"bm-verify|_Incapsula_|Client Challenge|px-captcha|captcha-delivery|cf-chl", re.I)


def price_forms(price):
    # "179.24" -> also "179,24"; "14.90" -> also "14.9"
    forms = {price, price.replace(".", ",")}
    if re.fullmatch(r"\d+\.\d0", price):
        forms.add(price[:-1])
    return forms


def check(url, shown_price):
    html = requests.get(url, headers=HEADERS, timeout=30).text
    scripts = re.findall(r"<script[^>]*>(.*?)</script>", html, flags=re.S | re.I)
    text = re.sub(r"<[^>]+>", " ", re.sub(r"<script.*?</script>|<style.*?</style>", " ", html, flags=re.S | re.I))

    if BOT_CHECK.search(html):
        return "bot check page: not a JavaScript problem"

    for block in re.findall(r'<script[^>]*application/ld\+json[^>]*>(.*?)</script>', html, flags=re.S | re.I):
        if re.search(r'"@type"\s*:\s*"(https?://schema\.org/)?Product"', block) and '"price"' in block:
            try:
                data = json.loads(block)
            except ValueError:
                continue
            return f"JSON-LD Product found: {json.dumps(data)[:120]}..."

    for form in price_forms(shown_price):
        pattern = r"(?<![\d.,])" + re.escape(form) + r"(?!\d)"
        if re.search(pattern, text):
            return f"price {form} found in the page text"
        if any(re.search(pattern, s) for s in scripts):
            return f"price {form} found in an embedded script (JSON)"

    if len(text.split()) < 50:
        return "almost no visible text: look for a bot check before rendering"
    return "price not in the HTML: this page needs JavaScript rendering"


if __name__ == "__main__":
    print(check(sys.argv[1], sys.argv[2]))

Run it as python check_page.py <url> 14.90. We ran it twice on eight pages on October 23, 2026, with the same answer both times: “JSON-LD Product found” for Nike and Snow+Rock, “price 14.9 found in an embedded script” for Uniqlo, “price 17.99 found in the page text” for Uncle Jeans, “bot check page” for Zara, “almost no visible text” for Nordstrom and Uncommon Goods, and “price not in the HTML” for Target.

Run it twice before you trust a negative answer. Earlier that day, one request to Uniqlo got a page without the price, and the next one, a few minutes later, had it.

Once you know where the price sits, the scraper is the short part: parse the JSON-LD or the embedded JSON with json.loads, or keep a selector for the text. Keep rendering for the pages that fail this check twice.

When you do need a browser to render JavaScript

In our plain requests from France, three pages showed no price: AS Discount, Uncommon Goods and Target. Rendering is the obvious next step, short of finding the request the page makes to its own API.

Search engines render pages too. Google describes three phases for JavaScript web apps, crawling, rendering and indexing, and says that “Once Google’s resources allow, a headless Chromium renders the page and executes the JavaScript” (Google Search Central, last updated March 4, 2026).

What JavaScript rendering changed in our test

We sent four pages to the Raw HTML endpoint, first with default settings, then with useChrome. On three of them, we ran a third call with waitForNetworkRequests added; it changed nothing.

Page	Default call	With useChrome
Uncommon Goods	$35.00 (US price), 3.3 s	$35.00, 22.4 s
Target	no price, 5.6 s	HTTP 504 after 60.1 s, twice
The Equine Warehouse	€45.00 in the meta tag, 2.0 s	the same €45.00, 6.1 s
AS Discount	179,24 €, 10.3 s	179,24 €, 7.6 s

Rendering returned no price that the default call had not already returned. Uncommon Goods came back in dollars both ways, while we see 38,00 € from France: on this page, the price followed the country of the request, so check the currency before you store a price. Target, the one page without a price in any response, ended in an HTTP 504 after 60.1 seconds on both rendering attempts, and we did not get its price through the API.

Selenium or a scraping API

If you run the browser yourself, Selenium with a headless Chrome is what many of the tutorials ranking for this topic teach, with Playwright as the other common choice. It runs on your machine, so it sees the site from your own connection and country, and you handle waits, crashes and memory. Selenium makes sense for a few pages seen from your own country. An API makes sense when the volume grows or when the price depends on the visitor’s country, as it did on Uncommon Goods.

To compare both modes on your own pages, paste a URL into the live test with a free account, or send it to the Raw HTML scraper API with and without rendering. The pricing page lists the monthly plans.

FAQ

Which is better for web scraping, Python or JavaScript?

For the check in this article, Python is the shorter path: a plain requests call was enough on 35 of the 40 pages that sent real content, with json for the structured data and a selector for the six prices in the text. Pick Node.js if your crawler already runs in JavaScript; how to build a web crawler uses it.

Is scraping a website illegal?

Collecting public product prices without logging in is the common case; personal data and anything behind a login raise more questions, and the answer also turns on the site’s terms and your country. This is not legal advice.