{"id":5397,"date":"2020-08-04T14:03:24","date_gmt":"2020-08-04T14:03:24","guid":{"rendered":"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-infinite-scroll-pages\/"},"modified":"2020-08-04T14:03:24","modified_gmt":"2020-08-04T14:03:24","slug":"how-to-scrape-infinite-scroll-pages","status":"publish","type":"post","link":"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-infinite-scroll-pages\/","title":{"rendered":"How to scrape Infinite Scroll Web Pages?"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"5397\" class=\"elementor elementor-5397\" data-elementor-post-type=\"post\">\n\t\t\t\t<div class=\"elementor-element elementor-element-86aaaae e-con-full e-flex e-con e-parent\" data-id=\"86aaaae\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-8eed32c elementor-widget elementor-widget-text-editor\" data-id=\"8eed32c\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<div style=\"height:50px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div> <div class=\"wp-block-image\"><figure class=\"alignleft is-resized\"><img fetchpriority=\"high\" decoding=\"async\" src=\"https:\/\/scraping-bot.io\/blogs\/wp-content\/uploads\/2026\/04\/Inf-2-1024x768-1.webp\" alt=\"Scraping infinite scroll pages\" class=\"wp-image-3919\" width=\"292\" height=\"219\"\/><\/figure><\/div> <p>Some pages are harder to scrape than other. In the case of infinite scrolling pages, scraping them will result in incomplete data collection.<\/p> <p>Infinite scrolling can be a way to prevent web scraping. You can read about other anti scraping methods in <a href=\"https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\"this article (opens in a new tab)\">this article<\/a>.<\/p> <div style=\"height:10px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-f711652 elementor-widget elementor-widget-heading\" data-id=\"f711652\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">What are Infinite Scroll Pages and what makes them harder to scrape?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-5363be7 elementor-widget elementor-widget-text-editor\" data-id=\"5363be7\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>If you scrape the webpage URL as it is, you will only collect the data of the first visible segment. It will also problematic for a crawler, as it won\ufffdt find the next pages URLs.&nbsp;<br><\/p> <p>When you visit this kind of webpage, the next content will automatically load when you reach the end of the page. One way to scrape the HTML of this kind of page can be by simulation of a human behavior with specific tools, such as Splash or Selenium. But there are more simple ways to do it.<br><\/p> <p>In reality, there is a pagination in an infinite scrolling page, but it is hidden in the HTML code. On a classic page, the user clicks on the next page URL whereas here, the next page is called dynamically when you visit the end area of the webpage.<\/p> <div style=\"height:20px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div> <h3>Infinite Scroll Page example:<\/h3> <p>We\ufffdre going to use this page as an example:<\/p> <figure class=\"wp-block-image size-full\"><img decoding=\"async\" src=\"https:\/\/scraping-bot.io\/blogs\/wp-content\/uploads\/2023\/08\/infinite-scroll-pages-example.jpg\" alt=\"Infinite scroll page example\" class=\"wp-image-9784\"\/><\/figure> <div style=\"height:20px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d2178ff elementor-widget elementor-widget-heading\" data-id=\"d2178ff\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">How do I find the URLs to scrape in the HTML code?<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-83e0aa2 elementor-widget elementor-widget-text-editor\" data-id=\"83e0aa2\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p>First, let\ufffds open the Developer Tools of your browser. For that, you just need to right click and select <em>Inspect Element<\/em>.&nbsp;<\/p> <p>Go in the Network tab. Most of the time, the requests we\ufffdre interested in will be in the XHR filter. On the webpage, scroll to the bottom to trigger the next page call. You will see in the network tab some requests being made by your browser. Click on one of them to see more details.&nbsp;<br><\/p> <p>Here are the requests we can see for this page:<\/p> <figure class=\"wp-block-image size-full\"><img decoding=\"async\" src=\"https:\/\/scraping-bot.io\/blogs\/wp-content\/uploads\/2023\/08\/requests-scraping-html.jpg\" alt=\"Requests scraping HTML\" class=\"wp-image-9796\"\/><\/figure> <p>If you don\ufffdt see any requests in the XHR tab, go back to <em>All<\/em>, and search for <em>Results<\/em>.<\/p> <p>As you can see, the browser is calling for the next pages.<br><\/p> <p>To scrape those pages, you will need to call them the same way your browser. So at the end of the URL, add <strong>\/results?page=1<\/strong><br><\/p> <p>So you can either scrape manually the URLs, or you can adapt your<a aria-label=\" web crawler (opens in a new tab)\" href=\"https:\/\/scraping-bot.io\/blogs\/how-to-build-a-web-crawler\/\" target=\"_blank\" rel=\"noreferrer noopener\"> web crawler<\/a> to increment the page number.&nbsp;For that, your crawler needs to generate requests to the URL and make it end by <strong>?page=X<\/strong>, until the request receives a 404 error, meaning the previous page was the last one.<\/p> <div style=\"height:20px\" aria-hidden=\"true\" class=\"wp-block-spacer\"><\/div> <div class=\"wp-block-buttons is-content-justification-center\"> <div class=\"wp-block-button has-custom-width wp-block-button__width-75 is-style-outline\"><a class=\"wp-block-button__link has-vivid-green-cyan-color has-text-color\" href=\"https:\/\/scraping-bot.io\/blogs\/pricing-web-scraper-api\/\">See pricing<\/a><\/div> <\/div>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Some pages are harder to scrape than other. In the case of infinite scrolling pages, scraping them will result in incomplete data collection. Infinite scrolling can be a way to prevent web scraping. You can read about other anti scraping methods in this article. What are Infinite Scroll Pages and what makes them harder to [&hellip;]<\/p>\n","protected":false},"author":0,"featured_media":5426,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[6],"tags":[],"class_list":["post-5397","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-web-scraping-in-general"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>How to scrape Infinite Scroll Web Pages? - Scraping-bot<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-infinite-scroll-pages\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"How to scrape Infinite Scroll Web Pages? - Scraping-bot\" \/>\n<meta property=\"og:description\" content=\"Some pages are harder to scrape than other. In the case of infinite scrolling pages, scraping them will result in incomplete data collection. Infinite scrolling can be a way to prevent web scraping. You can read about other anti scraping methods in this article. What are Infinite Scroll Pages and what makes them harder to [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-infinite-scroll-pages\/\" \/>\n<meta property=\"og:site_name\" content=\"Scraping-bot\" \/>\n<meta property=\"article:published_time\" content=\"2020-08-04T14:03:24+00:00\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/how-to-scrape-infinite-scroll-pages\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/how-to-scrape-infinite-scroll-pages\\\/\"},\"author\":{\"name\":\"\",\"@id\":\"\"},\"headline\":\"How to scrape Infinite Scroll Web Pages?\",\"datePublished\":\"2020-08-04T14:03:24+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/how-to-scrape-infinite-scroll-pages\\\/\"},\"wordCount\":433,\"publisher\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/how-to-scrape-infinite-scroll-pages\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/wp-content\\\/uploads\\\/2020\\\/08\\\/Infinite.webp\",\"articleSection\":[\"Web Scraping in general\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/how-to-scrape-infinite-scroll-pages\\\/\",\"url\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/how-to-scrape-infinite-scroll-pages\\\/\",\"name\":\"How to scrape Infinite Scroll Web Pages? - Scraping-bot\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/how-to-scrape-infinite-scroll-pages\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/how-to-scrape-infinite-scroll-pages\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/wp-content\\\/uploads\\\/2020\\\/08\\\/Infinite.webp\",\"datePublished\":\"2020-08-04T14:03:24+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/how-to-scrape-infinite-scroll-pages\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/how-to-scrape-infinite-scroll-pages\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/how-to-scrape-infinite-scroll-pages\\\/#primaryimage\",\"url\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/wp-content\\\/uploads\\\/2020\\\/08\\\/Infinite.webp\",\"contentUrl\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/wp-content\\\/uploads\\\/2020\\\/08\\\/Infinite.webp\",\"width\":900,\"height\":702},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/how-to-scrape-infinite-scroll-pages\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home &gt; Blog\",\"item\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"How to scrape Infinite Scroll Web Pages?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/#website\",\"url\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/\",\"name\":\"Scraping-bot\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/#organization\",\"name\":\"Scraping-bot\",\"url\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/wp-content\\\/uploads\\\/2025\\\/10\\\/scraping-bot-logo.svg\",\"contentUrl\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/wp-content\\\/uploads\\\/2025\\\/10\\\/scraping-bot-logo.svg\",\"width\":159,\"height\":32,\"caption\":\"Scraping-bot\"},\"image\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.linkedin.com\\\/company\\\/scrapingbot\\\/\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"How to scrape Infinite Scroll Web Pages? - Scraping-bot","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-infinite-scroll-pages\/","og_locale":"en_US","og_type":"article","og_title":"How to scrape Infinite Scroll Web Pages? - Scraping-bot","og_description":"Some pages are harder to scrape than other. In the case of infinite scrolling pages, scraping them will result in incomplete data collection. Infinite scrolling can be a way to prevent web scraping. You can read about other anti scraping methods in this article. What are Infinite Scroll Pages and what makes them harder to [&hellip;]","og_url":"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-infinite-scroll-pages\/","og_site_name":"Scraping-bot","article_published_time":"2020-08-04T14:03:24+00:00","twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-infinite-scroll-pages\/#article","isPartOf":{"@id":"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-infinite-scroll-pages\/"},"author":{"name":"","@id":""},"headline":"How to scrape Infinite Scroll Web Pages?","datePublished":"2020-08-04T14:03:24+00:00","mainEntityOfPage":{"@id":"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-infinite-scroll-pages\/"},"wordCount":433,"publisher":{"@id":"https:\/\/scraping-bot.io\/blogs\/#organization"},"image":{"@id":"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-infinite-scroll-pages\/#primaryimage"},"thumbnailUrl":"https:\/\/scraping-bot.io\/blogs\/wp-content\/uploads\/2020\/08\/Infinite.webp","articleSection":["Web Scraping in general"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-infinite-scroll-pages\/","url":"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-infinite-scroll-pages\/","name":"How to scrape Infinite Scroll Web Pages? - Scraping-bot","isPartOf":{"@id":"https:\/\/scraping-bot.io\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-infinite-scroll-pages\/#primaryimage"},"image":{"@id":"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-infinite-scroll-pages\/#primaryimage"},"thumbnailUrl":"https:\/\/scraping-bot.io\/blogs\/wp-content\/uploads\/2020\/08\/Infinite.webp","datePublished":"2020-08-04T14:03:24+00:00","breadcrumb":{"@id":"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-infinite-scroll-pages\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/scraping-bot.io\/blogs\/how-to-scrape-infinite-scroll-pages\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-infinite-scroll-pages\/#primaryimage","url":"https:\/\/scraping-bot.io\/blogs\/wp-content\/uploads\/2020\/08\/Infinite.webp","contentUrl":"https:\/\/scraping-bot.io\/blogs\/wp-content\/uploads\/2020\/08\/Infinite.webp","width":900,"height":702},{"@type":"BreadcrumbList","@id":"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-infinite-scroll-pages\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home &gt; Blog","item":"https:\/\/scraping-bot.io\/blogs\/"},{"@type":"ListItem","position":2,"name":"How to scrape Infinite Scroll Web Pages?"}]},{"@type":"WebSite","@id":"https:\/\/scraping-bot.io\/blogs\/#website","url":"https:\/\/scraping-bot.io\/blogs\/","name":"Scraping-bot","description":"","publisher":{"@id":"https:\/\/scraping-bot.io\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/scraping-bot.io\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/scraping-bot.io\/blogs\/#organization","name":"Scraping-bot","url":"https:\/\/scraping-bot.io\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/scraping-bot.io\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/scraping-bot.io\/blogs\/wp-content\/uploads\/2025\/10\/scraping-bot-logo.svg","contentUrl":"https:\/\/scraping-bot.io\/blogs\/wp-content\/uploads\/2025\/10\/scraping-bot-logo.svg","width":159,"height":32,"caption":"Scraping-bot"},"image":{"@id":"https:\/\/scraping-bot.io\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.linkedin.com\/company\/scrapingbot\/"]}]}},"_links":{"self":[{"href":"https:\/\/scraping-bot.io\/blogs\/wp-json\/wp\/v2\/posts\/5397","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scraping-bot.io\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scraping-bot.io\/blogs\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/scraping-bot.io\/blogs\/wp-json\/wp\/v2\/comments?post=5397"}],"version-history":[{"count":0,"href":"https:\/\/scraping-bot.io\/blogs\/wp-json\/wp\/v2\/posts\/5397\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/scraping-bot.io\/blogs\/wp-json\/wp\/v2\/media\/5426"}],"wp:attachment":[{"href":"https:\/\/scraping-bot.io\/blogs\/wp-json\/wp\/v2\/media?parent=5397"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scraping-bot.io\/blogs\/wp-json\/wp\/v2\/categories?post=5397"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scraping-bot.io\/blogs\/wp-json\/wp\/v2\/tags?post=5397"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}