Documentation

Have a question?

API Reference

Raw HTML Extraction Advanced Options Real Estate API Retail API Search Engine API Social Media API Booking API Build a web crawler

Build a web crawler

A web crawler automatically navigates a website by following links from page to page, building a complete map of its structure. Combined with ScrapingBot's API, it handles proxies, CAPTCHAs, and JavaScript rendering for every page it visits — so you can focus on collecting the data you need.

This guide walks through a production-ready crawler in Node.js using only two dependencies: request and cheerio. It uses a Breadth-First Search (BFS) strategy and respects rate limits with randomised delays between requests.

📦

Requirements: Node.js 8+ (for async/await support). Install dependencies with npm install request cheerio.

How it works

The crawler uses a BFS queue: starting from a seed URL, it fetches the page, extracts all links, adds unvisited ones to the queue, and repeats until the queue is empty or the maximum depth is reached.

Seed URL

Add the starting URL to the BFS queue at depth 0.

Fetch HTML

ScrapingBot fetches the page, bypassing proxies and CAPTCHAs.

Extract links

Parse all <a href> tags, filter same-domain URLs, deduplicate.

Queue & repeat

Add new links to the queue. Wait a random delay, then crawl next URL.

Configuration

All crawler settings are defined at the top of the file. Adjust these values before running.

Variable	Type	Default	Description
username	`string`	—	Your ScrapingBot username. Found in your dashboard.
apiKey	`string`	—	Your ScrapingBot API key. Found in your dashboard.
maxCrawlingDepth	`number`	5	Maximum link depth to crawl from the seed URL. See the depth guide below.
minimumWaitTime	`number (ms)`	500	Minimum delay between requests in milliseconds. Increase to at least 30 000 ms in production to avoid overloading the target website.
maximumWaitTime	`number (ms)`	5 000	Maximum delay between requests. The actual wait is randomised between min and max.
useChrome	`boolean`	false	Enable headless Chrome rendering for JavaScript-heavy pages. Consumes 10 credits per page instead of 1.
premiumProxy	`boolean`	false	Use residential proxies for heavily protected websites. Consumes 10 credits per page.

Understanding crawl depth

Depth controls how many levels of links the crawler follows from the seed URL. It grows exponentially — use with care on large websites.

Depth	Pages crawled	Use case
1	Seed page only + all links found on it	Quick scan of a single page's outbound links
2	All pages linked from depth 1	Mapping the top-level structure of a site
3	All pages linked from depth 2	Collecting product listings, articles, or directory pages
5	Potentially thousands of pages	Deep site audits — only on small or well-known sites
> 5	Tens of thousands of pages	Not recommended — very high credit consumption and risk of blocking

⚠️

Credit consumption: Each page crawled consumes at least 1 credit (10 with useChrome or premiumProxy). At depth 3 on a typical site, you may crawl 500–2 000 pages. Plan your credits accordingly.

Full source code

The complete crawler below is ready to run. Replace yourUsername and yourApiKey with your credentials, then call crawlBFS("https://your-target.com", 2).

crawler.js

const request = require("request");
const util    = require("util");
const cheerio = require("cheerio");
const { URL } = require("url");

const rp    = util.promisify(request);
const sleep = util.promisify(setTimeout);

// ─── State ───────────────────────────────────────────────────────────────────
let seenLinks    = {};
let rootNode     = {};
let currentNode  = {};
let linksQueue   = [];
let printList    = [];
let previousDepth = 0;

// ─── Configuration ───────────────────────────────────────────────────────────
let maxCrawlingDepth = 2;     // max depth — keep ≤ 5
let minimumWaitTime  = 500;   // ms — use ≥ 30000 in production
let maximumWaitTime  = 5000;  // ms

let mainDomain    = null;
let mainParsedUrl = null;

// ─── ScrapingBot credentials ─────────────────────────────────────────────────
const username    = "yourUsername";
const apiKey      = "yourApiKey";
const apiEndPoint = "https://api.scraping-bot.io/scrape/raw-html";
const auth        = "Basic " + Buffer.from(`${username}:${apiKey}`).toString("base64");

let requestOptions = {
  method : "POST",
  url    : apiEndPoint,
  json   : {
    url    : "", // replaced dynamically in findLinks()
    options: {
      useChrome   : false, // set true for JS-rendered pages (10 credits/page)
      premiumProxy: false, // set true for protected sites (10 credits/page)
    },
  },
  headers: {
    Accept       : "application/json",
    Authorization: auth,
  },
};

// ─── Link node ───────────────────────────────────────────────────────────────
class CreateLink {
  constructor(linkURL, depth, parent) {
    this.url      = linkURL;
    this.depth    = depth;
    this.parent   = parent;
    this.children = [];
  }
}

// ─── Entry point ─────────────────────────────────────────────────────────────
// Depth 1 = seed page + its direct links only
// Depth 2 = also follows links found on depth-1 pages
crawlBFS("https://www.scraping-bot.io/", 2);

async function crawlBFS(startURL, maxDepth = 5) {
  try {
    mainParsedUrl = new URL(startURL);
  } catch (e) {
    console.log("Invalid URL:", e);
    return;
  }
  mainDomain       = mainParsedUrl.hostname;
  maxCrawlingDepth = maxDepth;

  const startLinkObj = new CreateLink(startURL, 0, null);
  rootNode = currentNode = startLinkObj;
  addToLinkQueue(currentNode);
  await findLinks(currentNode);
}

async function crawl(linkObj) {
  await findLinks(linkObj);
}

// ─── Core: fetch page & extract links ────────────────────────────────────────
async function findLinks(linkObj) {
  requestOptions.json.url = linkObj.url;
  console.log(`Scraping [depth ${linkObj.depth}]: ${linkObj.url}`);

  let response;
  try {
    response = await rp(requestOptions);

    if (response.statusCode !== 200) {
      if (response.statusCode === 401) {
        console.log("Authentication failed — check your credentials.");
      } else {
        console.log(`Error ${response.statusCode} on ${linkObj.url}`);
      }
      return;
    }

    // ── Extract data here ────────────────────────────────────────────────────
    // response.body contains the full HTML of the page.
    // Add your own scraping logic below, e.g.:
    // const $ = cheerio.load(response.body);
    // const title = $("title").text();
    // ─────────────────────────────────────────────────────────────────────────

    const $ = cheerio.load(response.body);
    const links = $("body").find("a")
      .filter((i, el) => $(el).attr("href") != null)
      .map((i, el) => $(el).attr("href"))
      .get();

    if (links.length > 0) {
      links.forEach(href => {
        const resolvedURL = checkDomain(href);
        if (resolvedURL && resolvedURL !== linkObj.url) {
          const newLinkObj = new CreateLink(resolvedURL, linkObj.depth + 1, linkObj);
          addToLinkQueue(newLinkObj);
        }
      });
    } else {
      console.log(`No links found on ${linkObj.url}`);
    }

    const nextLinkObj = getNextInQueue();
    if (nextLinkObj && nextLinkObj.depth <= maxCrawlingDepth) {
      const waitTime = Math.round(
        minimumWaitTime + Math.random() * (maximumWaitTime - minimumWaitTime)
      );
      console.log(`Waiting ${waitTime}ms before next request…`);
      await sleep(waitTime);
      await crawl(nextLinkObj);
    } else {
      setRootNode();
      printTree();
      console.log("✅ Crawl complete.");
    }
  } catch (err) {
    console.log("Unexpected error:", err);
  }
}

// ─── Tree output ─────────────────────────────────────────────────────────────
function setRootNode() {
  while (currentNode.parent != null) currentNode = currentNode.parent;
  rootNode = currentNode;
}

function printTree() {
  addToPrintDFS(rootNode);
  console.log("\n── Crawl tree ──\n" + printList.join("\n"));
}

function addToPrintDFS(node) {
  const indent = "  ".repeat(node.depth);
  printList.push(`${indent}${node.depth > 0 ? "└─ " : ""}${node.url}`);
  if (node.children) node.children.forEach(child => addToPrintDFS(child));
}

// ─── URL helpers ─────────────────────────────────────────────────────────────
function checkDomain(linkURL) {
  let parsedUrl;
  try {
    parsedUrl = new URL(linkURL);
  } catch (_) {
    if (linkURL.startsWith("/")) {
      return `${mainParsedUrl.protocol}//${mainParsedUrl.hostname}${linkURL.split("#")[0]}`;
    } else if (linkURL.startsWith("#")) {
      return null; // anchor — skip
    } else {
      const path = currentNode.url.match(".*/")[0];
      return path + linkURL;
    }
  }
  if (parsedUrl.hostname === mainDomain) {
    parsedUrl.hash = "";
    return parsedUrl.href;
  }
  return null; // external domain — skip
}

function addToLinkQueue(linkObj) {
  if (!linkInSeenListExists(linkObj)) {
    if (linkObj.parent != null) linkObj.parent.children.push(linkObj);
    linksQueue.push(linkObj);
    addToSeen(linkObj);
  }
}

function getNextInQueue() {
  const nextLink = linksQueue.shift();
  if (nextLink && nextLink.depth > previousDepth) {
    previousDepth = nextLink.depth;
    console.log(`\n── Crawling depth ${nextLink.depth} ──`);
  }
  return nextLink;
}

function addToSeen(linkObj)           { seenLinks[linkObj.url] = linkObj; }
function linkInSeenListExists(linkObj) { return seenLinks[linkObj.url] != null; }

How to extract data while crawling

The crawler's findLinks() function receives the full HTML of every page in response.body. Add your extraction logic right after the HTML is received — before the link-following loop. Here's an example that collects page titles and meta descriptions:

extract-example.js — inside findLinks()

// Add this block right after: response = await rp(requestOptions)
const $ = cheerio.load(response.body);

const pageData = {
  url        : linkObj.url,
  depth      : linkObj.depth,
  title      : $("title").text().trim(),
  description: $('meta[name="description"]').attr("content") || null,
  h1         : $("h1").first().text().trim() || null,
  crawledAt  : new Date().toISOString(),
};

console.log(pageData);
// Or save to a JSON file, database, CSV, etc.

Best practices & rules

🚫

Always respect robots.txt before crawling any website. Check https://target-site.com/robots.txt to see which paths are allowed for bots. Ignoring it may result in legal consequences.

⏱️

Use realistic delays. The default minimumWaitTime of 500 ms is only suitable for testing. In production, use at least 30 000 ms (30 seconds) between requests to avoid overloading the target server and getting your IP blocked.

✅

Start shallow. Always start with maxDepth = 1 to validate your extraction logic on a small number of pages before increasing the depth. This saves credits and avoids unintended mass crawling.

💡

Use useChrome: true selectively. Only enable headless Chrome for pages you know require JavaScript rendering. Since it costs 10× more credits, try the standard mode first.

Rule	Why it matters
Check robots.txt	Legal compliance and avoiding bans
Keep delays ≥ 30s in production	Prevents server overload and IP blocking
Limit depth to ≤ 5	Avoids exponential page growth and excessive credit use
Stay on the same domain	The crawler already filters external links — keep this behaviour
Deduplicate URLs	The seenLinks set prevents crawling the same page twice
Handle errors gracefully	Log failed requests and continue — don't let one error stop the whole crawl

Works with the best tools: