The n8n HTTP Request node is all you need to turn the ScrapingBot API into a no-code scraping engine: no custom scripts, no proxy pool to babysit, no headless browser to maintain. This guide wires the two together correctly, from Basic Auth setup to reading the response, scraping hundreds of URLs, and handling the errors that actually show up in production.
Quick answer: Connect n8n to the ScrapingBot API with an HTTP Request node authenticated via Basic Auth: set Response Format to Text and enable Include Response Headers and Status, since raw-html returns HTML in the body field, not JSON. Turn on useChrome for JavaScript-heavy pages and premiumProxy for protected sites, batch 5 to 10 URLs at a time with a short delay between groups, and check the real HTTP status code to catch failures.
Table of contents
1. Why pair n8n with the ScrapingBot API
A scraping pipeline needs two very different jobs done well: pulling clean data off a page, and orchestrating what happens to that data next. Most teams glue those jobs together with fragile scripts and cron files. n8n and ScrapingBot split the work cleanly instead, so each side does what it is good at.
| Tool | What it handles |
|---|---|
| n8n | Visual workflow builder: triggers, branching, batching, scheduling, and 400+ app integrations. |
| ScrapingBot | Scraping API: JavaScript rendering, residential proxies, geolocation, and automatic CAPTCHA handling. |
Together they cover the whole path, from scraping a page to writing rows in a database, posting a Slack alert, or updating a Google Sheet. For example, a marketer can build a price watcher without touching a proxy config. In practice, that is why the combination fits teams who want reliable data collection without owning scraping infrastructure.
2. What you need before you start
Three things, and you can have all of them in a few minutes:
- An n8n instance, either n8n Cloud, a self-hosted server, or the desktop app.
- Your ScrapingBot username and API key, both shown in your dashboard.
- A target URL you want to scrape.
The free plan gives you 1,000 credits per month with no card required, which is enough to build and test a full workflow before you commit to anything.
3. Configure the n8n HTTP Request node
Create a reusable Basic Auth credential
ScrapingBot authenticates with HTTP Basic Auth, so store your keys once instead of pasting them into every node. However, keep them in a credential rather than the request body, which keeps the key out of your exported workflows.
- In n8n, open Credentials, then Add Credential.
- Choose Basic Auth.
- Name it ScrapingBot API.
- Set User to your ScrapingBot username.
- Set Password to your ScrapingBot API key, then save.
Set up the node
Add an HTTP Request node and match the settings below. The response format is the part people get wrong, so read the last two rows carefully.
| Field | Value |
|---|---|
| Method | POST |
| URL | https://api.scraping-bot.io/scrape/raw-html |
| Authentication | Generic Credential, then Basic Auth, then select ScrapingBot API |
| Send Body | On, Body Content Type set to JSON |
| Response Format (in Options) | Text, because raw-html returns HTML, not JSON |
| Include Response Headers and Status | On, so you can read the real HTTP status for error checks |
Set the request body
Pass the target URL plus any options. The three options that matter most are useChrome, premiumProxy, and waitForNetworkRequests:
{
"url": "https://example.com/products",
"options": {
"useChrome": true,
"premiumProxy": false,
"waitForNetworkRequests": true
}
}Here is what each option does, and none of this is guesswork, it maps directly to the ScrapingBot documentation:
- useChrome renders JavaScript with a headless Chrome. Turn this on for React, Vue, or Angular pages. This is the real fix for an empty or half-loaded page, not a wait setting.
- premiumProxy routes the request through residential IPs for well-defended sites such as Amazon, Google, or LinkedIn.
- waitForNetworkRequests waits for AJAX and XHR calls to finish before returning the HTML. It only works when useChrome is true.
- proxyCountry targets a country-specific proxy, for example US or FR. It requires premiumProxy set to true.
For geolocation, combine premiumProxy with proxyCountry. Leaving premiumProxy off makes proxyCountry a no-op, which is a common silent mistake:
{
"url": "https://www.amazon.com/dp/B0EXAMPLE",
"options": {
"premiumProxy": true,
"proxyCountry": "US"
}
}When the URL comes from a previous node, a Google Sheets row for instance, swap the static value for an n8n expression:
{
"url": "{{ $json.url }}",
"options": { "useChrome": true }
}Options are not free, so it helps to know the cost before you flip everything on. Credits are charged per request as follows:
| Options used | Credits per request |
|---|---|
| Standard, no options | 1 |
| useChrome | 10 |
| premiumProxy | 10 |
| useChrome + premiumProxy | 20 |
| proxyCountry (with premiumProxy) | 25 |
4. Read the response: raw HTML is text, not JSON
This is where most n8n tutorials, including the earlier version of this one, go wrong. The raw-html endpoint returns the fully rendered page as a plain HTML string. It does not return a JSON object with fields like statusCode, captchaFound, or host. Those fields belong to the structured APIs, Retail, Real Estate, and Booking, not to raw-html.
Because you enabled Include Response Headers and Status, the node output looks like this, with the HTML sitting under the body property:
{
"statusCode": 200,
"body": "<html>...fully rendered page...</html>",
"headers": { ... }
}To pull fields out of that HTML, add an HTML node (older versions call it HTML Extract) after the request. Point its source at the body property and describe what you want with CSS selectors:
| Output field | CSS selector | Return value |
|---|---|---|
| productTitle | h2.product-title | Text |
| productPrice | span.price | Text |
| productUrl | a.product-link | Attribute, href |
One shortcut worth knowing: if your target is a supported e-commerce, real estate, or hotel site, skip the HTML parsing entirely. Point the node at the Retail, Real Estate, or Booking endpoint instead, and ScrapingBot returns clean JSON with price, currency, and stock already extracted. That is the case where statusCode and captchaFound genuinely appear in the response.
5. Scrape many URLs without getting blocked
Firing a hundred requests at once is the fastest way to trip rate limits and hammer the target server. Batching solves both problems. The pattern below reads a list of URLs, processes them in small groups, and adds a short pause between each group.
| Step | Node | Purpose |
|---|---|---|
| 1 | Trigger (Manual or Schedule) | Start the workflow |
| 2 | Google Sheets or Database | Read the list of URLs |
| 3 | Loop Over Items (Split in Batches) | Process 5 to 10 URLs at a time |
| 4 | HTTP Request to ScrapingBot | Scrape each URL |
| 5 | Wait | Pause 500 to 1500 ms between groups |
| 6 | HTML or Code | Parse the response body |
| 7 | Write results | Push to a sheet, database, or CRM |
For the pause, a random delay looks more natural than a fixed one. Drop a Code node before the Wait node:
// random delay between 500 and 1500 ms
const delay = Math.floor(Math.random() * 1000) + 500;
return [{ json: { delay } }];Then set the Wait node duration to the expression {{ $json.delay }} in milliseconds. As a result, the workflow paces itself like a human browsing session and is far less likely to get throttled.
6. Handle errors the right way
Since raw-html hands back an HTTP status on the response itself, you check that status rather than a made-up JSON flag. Add an IF node after the request and test the real status plus a non-empty body:
{{ $json.statusCode === 200 && $json.body.length > 0 }}Route the false branch to a retry or an error handler. These are the failures you will meet most often, and their real fixes:
| Symptom | Cause | Fix |
|---|---|---|
| 401 Unauthorized | Wrong credentials | Recheck the username and API key in the n8n credential |
| 429 Too Many Requests | Rate limit hit | Raise the delay or shrink the batch size |
| Empty or partial HTML | JavaScript not rendered | Set useChrome to true, add waitForNetworkRequests |
| Blocked or CAPTCHA page | Tough anti-bot target | Add premiumProxy to true for residential IPs |
| 500 from target | Target site error | Retry with a short backoff |
| Timeout | Heavy page or oversized batch | Smaller batches, add a Wait node, requests cap at 60s |
For transient failures, let the node retry itself. In the HTTP Request settings, enable Retry On Fail, set Max Tries to 3, and Wait Between Tries to 2000 ms. For anything that keeps failing, send the URL to a dedicated branch that logs it to a sheet or pings you on Slack for a manual look.
7. Three production-ready workflows
Once the basic flow runs, these three automations are small variations on it that you can ship today.
Price monitor
Track a product price and get pinged when it moves. A Schedule trigger runs hourly, the HTTP Request scrapes the product page, an HTML node reads the current price, an IF node compares it with the last stored value, and a Slack or Email node fires only when it changed. A Google Sheets node then saves the new price for the next run.
Lead capture pipeline
Turn a list of company websites into enriched CRM records. Read the URLs from a sheet, loop over them in small batches, scrape each page, then use a Code node to pull the company name, email, phone, and address. A HubSpot or Salesforce node creates or updates the contact. One caveat: LinkedIn, Instagram, and other social profiles are not raw-html jobs. Those need ScrapingBot's dedicated Social Media API, which uses a two-step queue-and-poll flow.
SEO audit
Audit an entire site for missing titles and broken headings. Fetch your sitemap.xml, extract every URL with an XML node, and loop over them in groups of ten. For each page the HTTP Request returns both the statusCode and the HTML body, so an HTML node can pull the title, H1, and meta description while the status flags any page that is not returning 200. Export the whole thing to a Google Sheet.
8. FAQ
Do I need to write code to use n8n with ScrapingBot?
No. The n8n HTTP Request node sends the API call for you, and the HTML node parses the result with CSS selectors. A Code node is optional, useful only when you want custom logic like a random delay.
Which ScrapingBot endpoint should I call from n8n?
Use raw-html when you want the page HTML and will parse it yourself. Use the Retail, Real Estate, Booking, Search Engine, or Social Media endpoints when you want clean structured JSON without writing selectors.
Why is my scraped HTML empty?
The page most likely renders its content with JavaScript. Set useChrome to true, and add waitForNetworkRequests to true so the API waits for AJAX calls to finish. If the page is actively blocking you, add premiumProxy to true as well.
How many credits does one request cost?
A standard request costs 1 credit. useChrome or premiumProxy cost 10 each and both together cost 20. The free plan includes 1,000 credits per month.
Going further
You now have an accurate, production-ready template: correct options, the right way to read a raw HTML response, safe batching, and error handling built on the real HTTP status. Clone the workflow, point it at your own targets, and let n8n handle the orchestration while ScrapingBot handles the hard part.
