{"id":5390,"date":"2020-02-14T11:34:11","date_gmt":"2020-02-14T11:34:11","guid":{"rendered":"https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/"},"modified":"2020-02-14T11:34:11","modified_gmt":"2020-02-14T11:34:11","slug":"anti-scraping-methods","status":"publish","type":"post","link":"https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/","title":{"rendered":"What are the methods used against web scraping?"},"content":{"rendered":"\t\t<div data-elementor-type=\"wp-post\" data-elementor-id=\"5390\" class=\"elementor elementor-5390\" data-elementor-post-type=\"post\">\n\t\t\t\t<div class=\"elementor-element elementor-element-c42ab62 e-con-full e-flex e-con e-parent\" data-id=\"c42ab62\" data-element_type=\"container\" data-e-type=\"container\">\n\t\t\t\t<div class=\"elementor-element elementor-element-50f777e elementor-widget elementor-widget-text-editor\" data-id=\"50f777e\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p><\/p> <\/p> <figure><img fetchpriority=\"high\" decoding=\"async\" src=\"https:\/\/scraping-bot.io\/blogs\/wp-content\/uploads\/2020\/02\/scrapingbot-methods-used-against-webscraping.webp\" alt=\"Scrapingbot - Methods used to prevent webscraping\" width=\"326\" height=\"190\"><\/figure> <p> <\/p> <p>Even though web scraping is commonly used across most industries, most websites do not appreciate it and new anti-scraping methods are being developed regularly. The main reason is that aggressive web scraping can slow down the website for regular users, and in the worst-case result in a denial of service. To prevent you from scraping their websites, companies are using various strategies.<\/p> <p> <\/p> &nbsp; <p> <\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-fc389be elementor-widget elementor-widget-heading\" data-id=\"fc389be\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Limiting the scraping<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-7d21ac7 elementor-widget elementor-widget-text-editor\" data-id=\"7d21ac7\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p> <\/p> <p>IP rate limiting, also called requests throttling, is a commonly used anti-scraping method. A good practice of web scraping is to respect the website and scrape it slowly. This way, you will avoid monopolizing the bandwidth of the website. The goal is for the regular users to still have a smooth experience of the website in parallel of your scraping. IP rate limitation means that there is a maximum number of actions doable in a certain time on the website. Any request over this limit will simply not receive an answer.<\/p> <p> <\/p> &nbsp; <p> <\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d5a10c1 elementor-widget elementor-widget-heading\" data-id=\"d5a10c1\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Blocking the web scraping<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-48c1090 elementor-widget elementor-widget-text-editor\" data-id=\"48c1090\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p> <\/p> <p>While some websites are okay with a simple regulation of the web scraping, others are trying to prevent it all together. They are using many technics to detect and block scrapers: user agent, CAPTCHAs, behavioral analysis technology, blocking individual or entire IP ranges, AWS shield, \ufffd You can read more about how to scrape a website without being blocked in <a href=\"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-a-website-without-getting-blocked\/\">this article<\/a>.<\/p> <p> <\/p> &nbsp; <p> <\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-d3588f8 elementor-widget elementor-widget-heading\" data-id=\"d3588f8\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Making the data collection harder<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-a7b2203 elementor-widget elementor-widget-text-editor\" data-id=\"a7b2203\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p> <\/p> <p>Some websites modify their HTML markups every month to protect their data. A scraping bot will look for an information in the places it found it last time. By changing the pattern of their HTML, the websites are trying to confuse the scraping tool, and making it harder to find the desired data.<\/p> <p> <\/p> <p>In addition, the programmers can obfuscate the code. HTML obfuscation consist in making the code much harder to read, while keeping it perfectly functional. The information is still there but written in an extremely complex way.<\/p> <p> <\/p> <p>Another technique is to make a dynamic user interface with Javascript or AJAX. The page only loads some portions of the contents. The information to collect can be found behind some buttons, not requiring reloading the page. This will result in a time out when scraped.<\/p> <p> <\/p> &nbsp; <p> <\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-e833bf7 elementor-widget elementor-widget-heading\" data-id=\"e833bf7\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"heading.default\">\n\t\t\t\t\t<h2 class=\"elementor-heading-title elementor-size-default\">Providing fake information<\/h2>\t\t\t\t<\/div>\n\t\t\t\t<div class=\"elementor-element elementor-element-3a33547 elementor-widget elementor-widget-text-editor\" data-id=\"3a33547\" data-element_type=\"widget\" data-e-type=\"widget\" data-widget_type=\"text-editor.default\">\n\t\t\t\t\t\t\t\t\t<p> <\/p> <p>In our article about <a href=\"https:\/\/scraping-bot.io\/blogs\/how-to-scrape-a-website-without-getting-blocked\/\">scraping without getting blocked<\/a>, we\ufffdve talked about Honeypots, those links that only bots will find and visit. Some other techniques are also intended to be seen only by bots and not by regular users. This is the case of cloaking. This is a hiding technique that returns an altered page when visited by a bot. Normal user only will be able to see the real pages. &nbsp;The bot will still collect information, without knowing that it is fake or incorrect. This method is really frowned upon by Google and other search engines. Websites using it are taking the risk to be removed from their index.<\/p> <p> <\/p> &nbsp; <p> <\/p> <hr> <p> <\/p> <br> <p> <p><\/p>\t\t\t\t\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t\t\t<\/div>\n\t\t","protected":false},"excerpt":{"rendered":"<p>Even though web scraping is commonly used across most industries, most websites do not appreciate it and new anti-scraping methods are being developed regularly. The main reason is that aggressive web scraping can slow down the website for regular users, and in the worst-case result in a denial of service. To prevent you from scraping [&hellip;]<\/p>\n","protected":false},"author":0,"featured_media":5419,"comment_status":"","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"footnotes":""},"categories":[6],"tags":[],"class_list":["post-5390","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-web-scraping-in-general"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO plugin v27.4 - https:\/\/yoast.com\/product\/yoast-seo-wordpress\/ -->\n<title>What are the methods used against web scraping? - Scraping-bot<\/title>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"What are the methods used against web scraping? - Scraping-bot\" \/>\n<meta property=\"og:description\" content=\"Even though web scraping is commonly used across most industries, most websites do not appreciate it and new anti-scraping methods are being developed regularly. The main reason is that aggressive web scraping can slow down the website for regular users, and in the worst-case result in a denial of service. To prevent you from scraping [&hellip;]\" \/>\n<meta property=\"og:url\" content=\"https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/\" \/>\n<meta property=\"og:site_name\" content=\"Scraping-bot\" \/>\n<meta property=\"article:published_time\" content=\"2020-02-14T11:34:11+00:00\" \/>\n<meta property=\"og:image\" content=\"https:\/\/scraping-bot.io\/blogs\/wp-content\/uploads\/2020\/02\/scrapingbot-methods-used-against-webscraping.png\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:label1\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data1\" content=\"2 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/anti-scraping-methods\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/anti-scraping-methods\\\/\"},\"author\":{\"name\":\"\",\"@id\":\"\"},\"headline\":\"What are the methods used against web scraping?\",\"datePublished\":\"2020-02-14T11:34:11+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/anti-scraping-methods\\\/\"},\"wordCount\":488,\"publisher\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/anti-scraping-methods\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/wp-content\\\/uploads\\\/2020\\\/02\\\/scrapingbot-methods-used-against-webscraping.webp\",\"articleSection\":[\"Web Scraping in general\"],\"inLanguage\":\"en-US\"},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/anti-scraping-methods\\\/\",\"url\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/anti-scraping-methods\\\/\",\"name\":\"What are the methods used against web scraping? - Scraping-bot\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/anti-scraping-methods\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/anti-scraping-methods\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/wp-content\\\/uploads\\\/2020\\\/02\\\/scrapingbot-methods-used-against-webscraping.webp\",\"datePublished\":\"2020-02-14T11:34:11+00:00\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/anti-scraping-methods\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/anti-scraping-methods\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/anti-scraping-methods\\\/#primaryimage\",\"url\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/wp-content\\\/uploads\\\/2020\\\/02\\\/scrapingbot-methods-used-against-webscraping.webp\",\"contentUrl\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/wp-content\\\/uploads\\\/2020\\\/02\\\/scrapingbot-methods-used-against-webscraping.webp\",\"width\":481,\"height\":280},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/anti-scraping-methods\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Home &gt; Blog\",\"item\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"What are the methods used against web scraping?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/#website\",\"url\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/\",\"name\":\"Scraping-bot\",\"description\":\"\",\"publisher\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/#organization\"},\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/#organization\",\"name\":\"Scraping-bot\",\"url\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/wp-content\\\/uploads\\\/2025\\\/10\\\/scraping-bot-logo.svg\",\"contentUrl\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/wp-content\\\/uploads\\\/2025\\\/10\\\/scraping-bot-logo.svg\",\"width\":159,\"height\":32,\"caption\":\"Scraping-bot\"},\"image\":{\"@id\":\"https:\\\/\\\/scraping-bot.io\\\/blogs\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.linkedin.com\\\/company\\\/scrapingbot\\\/\"]}]}<\/script>\n<!-- \/ Yoast SEO plugin. -->","yoast_head_json":{"title":"What are the methods used against web scraping? - Scraping-bot","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/","og_locale":"en_US","og_type":"article","og_title":"What are the methods used against web scraping? - Scraping-bot","og_description":"Even though web scraping is commonly used across most industries, most websites do not appreciate it and new anti-scraping methods are being developed regularly. The main reason is that aggressive web scraping can slow down the website for regular users, and in the worst-case result in a denial of service. To prevent you from scraping [&hellip;]","og_url":"https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/","og_site_name":"Scraping-bot","article_published_time":"2020-02-14T11:34:11+00:00","og_image":[{"url":"https:\/\/scraping-bot.io\/blogs\/wp-content\/uploads\/2020\/02\/scrapingbot-methods-used-against-webscraping.png","type":"","width":"","height":""}],"twitter_card":"summary_large_image","twitter_misc":{"Est. reading time":"2 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/#article","isPartOf":{"@id":"https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/"},"author":{"name":"","@id":""},"headline":"What are the methods used against web scraping?","datePublished":"2020-02-14T11:34:11+00:00","mainEntityOfPage":{"@id":"https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/"},"wordCount":488,"publisher":{"@id":"https:\/\/scraping-bot.io\/blogs\/#organization"},"image":{"@id":"https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/#primaryimage"},"thumbnailUrl":"https:\/\/scraping-bot.io\/blogs\/wp-content\/uploads\/2020\/02\/scrapingbot-methods-used-against-webscraping.webp","articleSection":["Web Scraping in general"],"inLanguage":"en-US"},{"@type":"WebPage","@id":"https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/","url":"https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/","name":"What are the methods used against web scraping? - Scraping-bot","isPartOf":{"@id":"https:\/\/scraping-bot.io\/blogs\/#website"},"primaryImageOfPage":{"@id":"https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/#primaryimage"},"image":{"@id":"https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/#primaryimage"},"thumbnailUrl":"https:\/\/scraping-bot.io\/blogs\/wp-content\/uploads\/2020\/02\/scrapingbot-methods-used-against-webscraping.webp","datePublished":"2020-02-14T11:34:11+00:00","breadcrumb":{"@id":"https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/#primaryimage","url":"https:\/\/scraping-bot.io\/blogs\/wp-content\/uploads\/2020\/02\/scrapingbot-methods-used-against-webscraping.webp","contentUrl":"https:\/\/scraping-bot.io\/blogs\/wp-content\/uploads\/2020\/02\/scrapingbot-methods-used-against-webscraping.webp","width":481,"height":280},{"@type":"BreadcrumbList","@id":"https:\/\/scraping-bot.io\/blogs\/anti-scraping-methods\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Home &gt; Blog","item":"https:\/\/scraping-bot.io\/blogs\/"},{"@type":"ListItem","position":2,"name":"What are the methods used against web scraping?"}]},{"@type":"WebSite","@id":"https:\/\/scraping-bot.io\/blogs\/#website","url":"https:\/\/scraping-bot.io\/blogs\/","name":"Scraping-bot","description":"","publisher":{"@id":"https:\/\/scraping-bot.io\/blogs\/#organization"},"potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/scraping-bot.io\/blogs\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/scraping-bot.io\/blogs\/#organization","name":"Scraping-bot","url":"https:\/\/scraping-bot.io\/blogs\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/scraping-bot.io\/blogs\/#\/schema\/logo\/image\/","url":"https:\/\/scraping-bot.io\/blogs\/wp-content\/uploads\/2025\/10\/scraping-bot-logo.svg","contentUrl":"https:\/\/scraping-bot.io\/blogs\/wp-content\/uploads\/2025\/10\/scraping-bot-logo.svg","width":159,"height":32,"caption":"Scraping-bot"},"image":{"@id":"https:\/\/scraping-bot.io\/blogs\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.linkedin.com\/company\/scrapingbot\/"]}]}},"_links":{"self":[{"href":"https:\/\/scraping-bot.io\/blogs\/wp-json\/wp\/v2\/posts\/5390","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/scraping-bot.io\/blogs\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/scraping-bot.io\/blogs\/wp-json\/wp\/v2\/types\/post"}],"replies":[{"embeddable":true,"href":"https:\/\/scraping-bot.io\/blogs\/wp-json\/wp\/v2\/comments?post=5390"}],"version-history":[{"count":0,"href":"https:\/\/scraping-bot.io\/blogs\/wp-json\/wp\/v2\/posts\/5390\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/scraping-bot.io\/blogs\/wp-json\/wp\/v2\/media\/5419"}],"wp:attachment":[{"href":"https:\/\/scraping-bot.io\/blogs\/wp-json\/wp\/v2\/media?parent=5390"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/scraping-bot.io\/blogs\/wp-json\/wp\/v2\/categories?post=5390"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/scraping-bot.io\/blogs\/wp-json\/wp\/v2\/tags?post=5390"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}