How to scrape with Postman?

1. Why use Postman to call the ScrapingBot API

Postman turns any REST API into a set of clickable requests. There's no server to run and no client library to install, which is exactly what you want the first time you scrape with Postman: build the request once, hit Send, and read the result right there in the app.

That said, Postman is a testing and prototyping tool, not a production pipeline. It's the right choice for verifying a request works, pulling a handful of pages manually, or showing a teammate how an endpoint behaves before they wire it into real code. For anything scheduled or high-volume, you'll eventually want to move the same request into a script, but Postman is where most people should start.

2. What you need before you start

Three things, and all three take a few minutes to get:

• The free Postman desktop app, or its web version.
• A ScrapingBot account, which gives you a username and an API key from your dashboard.
• A public URL you want to scrape.

The free plan includes 500 credits a month with no card required, which is more than enough to work through every example in this guide.

3. Set up Basic Auth in Postman

Every ScrapingBot endpoint authenticates with HTTP Basic Auth, so this step only needs doing once per request collection.

1. Create a new request and open the Authorization tab.
2. Set Type to Basic Auth.
3. Enter your ScrapingBot username in the Username field.
4. Enter your API key in the Password field.

Postman encodes both values into the Authorization header for you, so you never have to base64-encode anything by hand.

4. Your first request: raw HTML extraction

The raw-html endpoint is the most flexible one: point it at any public page and it hands back the fully rendered HTML as plain text, which you can then parse however you like.

In the Body tab, select raw and JSON, then add the target URL:

{
  "url": "https://example.com"
}

Click Send and the response panel fills with the page's HTML. If the page you're targeting loads its content with JavaScript, a plain request may come back mostly empty, so add an options object with useChrome set to true:

{
  "url": "https://example.com",
  "options": {
    "useChrome": true
  }
}

useChrome renders the page with a real headless Chrome before returning the HTML, which fixes most empty-page problems on React, Vue, or Angular sites. It costs 10 credits per request instead of 1, so turn it on only when the plain request comes back incomplete.

Two more options are worth knowing. premiumProxy routes the request through a residential IP, useful on sites like Amazon or LinkedIn that block obvious datacenter traffic, and also costs 10 credits. proxyCountry targets a specific country's proxy pool for region-locked content, but only works alongside premiumProxy and raises the cost to 25 credits.

5. Skip the parsing with Retail and Real Estate

Raw HTML is flexible, but you still have to write selectors to pull out a price or a title. For supported sites, two specialized endpoints do that parsing for you and return clean JSON instead.

The retail endpoint covers Amazon, eBay, Walmart, Etsy, Zalando, and dozens of other stores. Send it a product URL and it returns fields like title, price, currency, images, inStock, brand, and rating, already extracted.

{
  "url": "https://www.amazon.com/dp/B0EXAMPLE"
}

Point that same JSON body at https://api.scraping-bot.io/scrape/retail instead of raw-html, and there's nothing left to parse. The real-estate endpoint works the same way for listing sites like Zillow, Rightmove, or SeLoger, returning price, surface area, number of rooms, and energy class where the portal provides one. JS rendering and proxy rotation are switched on automatically for both, so you don't need to add useChrome yourself.

6. Scraping social media: the two-step flow

Social platforms fight bots harder than most retail sites, so the social media endpoint splits the work into two calls instead of one. This avoids timeouts and lets ScrapingBot retry quietly in the background.

First, send a POST to https://api.scraping-bot.io/scrape/data-scraper naming the scraper you want and the target:

{
  "scraper": "linkedinProfile",
  "url": "https://www.linkedin.com/in/example"
}

The response returns a responseId immediately, before the scrape has actually finished. Wait at least 5 seconds, then send a GET request to retrieve the result:

GET https://api.scraping-bot.io/scrape/data-scraper-response?responseId=xxxx&scraper=linkedinProfile

If the scrape isn't ready yet, the response comes back with a pending status and you simply try again a few seconds later. Don't poll more often than every 5 seconds, since social scraping takes longer than a standard request and polling too fast just wastes calls. Instagram, Facebook, Twitter/X, TikTok, and Threads all use the same two-step pattern, each with their own scraper names and a handful of platform-specific parameters like posts_number or hashtag.

7. Common errors and how to fix them

Most problems trace back to one of a few causes, and Postman's response panel usually tells you which:

For the retail, real-estate, and social media endpoints specifically, a failed scrape still returns HTTP 200, so check the error field inside the JSON body rather than relying on the status code alone.

8. FAQ

Going further

You now have everything needed to scrape with Postman: Basic Auth configured once, a working raw HTML request, two structured endpoints that skip parsing entirely, and the two-step pattern social platforms require. Save each request into a Postman collection once it works, and you'll have a reusable reference the next time you need data from a new site.