Web scraping, from your first request to production.
Learning tracks, site-by-site guides and runnable code in Python, Node.js, PHP, Go, Java and Rust — plus independent tests of the infrastructure you will eventually need.
Learning path
Three stages, from first request to staying unblocked
Start with your language, then handle JavaScript-heavy sites, then keep your scraper from getting blocked.
- 01 · Foundations HTTP, HTML and selectors Your first scraper in Python, Node.js, PHP, Go, Java or Rust, then pagination.
- 02 · Dynamic sites JavaScript-rendered pages Playwright for JavaScript-rendered pages, or a hosted scraping browser.
- 03 · Staying unblocked Why scrapers get blocked Proxy types, rotation, CAPTCHAs, geo-targeting and avoiding blocks.
Techniques
Guides worth reading first
The techniques behind every scraper that keeps working: rotation, unblocking, pagination and the right proxy type.
AI and agents
Give AI agents live web data
How the scraping providers' MCP servers connect Claude, Cursor and other agents to the web, and which proxies suit AI data collection. Written from each provider's docs.
Tools
Practise and plan in your browser
Free games and calculators for the skills scraping depends on: writing selectors, staying unblocked and pricing a project. Nothing you enter leaves your browser.
Code first
Runnable code in your language
Learning guides cover Python, Node.js, PHP, Go, Java and Rust. Every site solution ships with code in PHP, Node.js and Rust.
- Copy-paste ready, no hidden helpers
- Same task, idiomatic in each language
- 20 site solutions: Amazon, Google, Zillow, Booking.com and more
import httpx from selectolax.parser import HTMLParser r = httpx.get("https://books.toscrape.com/", timeout=20) r.raise_for_status() for book in HTMLParser(r.text).css("article.product_pod"): title = book.css_first("h3 a").attributes["title"] price = book.css_first(".price_color").text() print(title, price)
import * as cheerio from "cheerio"; const res = await fetch("https://books.toscrape.com/"); const $ = cheerio.load(await res.text()); $("article.product_pod").each((_, el) => { const title = $(el).find("h3 a").attr("title"); const price = $(el).find(".price_color").text(); console.log(title, price); });
<?php require 'vendor/autoload.php'; use Symfony\Component\DomCrawler\Crawler; $html = (new GuzzleHttp\Client())->get('https://books.toscrape.com/')->getBody(); (new Crawler((string) $html))->filter('article.product_pod')->each(function ($n) { echo $n->filter('h3 a')->attr('title'), ' ', $n->filter('.price_color')->text(), PHP_EOL; });
use scraper::{Html, Selector}; fn main() -> Result<(), Box<dyn std::error::Error>> { let body = reqwest::blocking::get("https://books.toscrape.com/")?.text()?; let doc = Html::parse_document(&body); let card = Selector::parse("article.product_pod").unwrap(); let title = Selector::parse("h3 a").unwrap(); let price = Selector::parse(".price_color").unwrap(); for el in doc.select(&card) { let t = el.select(&title).next().and_then(|a| a.value().attr("title")).unwrap_or(""); let p = el.select(&price).next().map(|p| p.text().collect::<String>()).unwrap_or_default(); println!("{t} {p}"); } Ok(()) }
For companies
Planning a data collection project?
Not every team should build scrapers. Start with the use case, then decide whether to build, use a no-code tool or buy the data.
Provider rankings
When you outgrow DIY
Our top-rated proxy providers and scraping APIs. Each review explains how its score was reached.
Proxy providers
All proxy rankings →| # | Provider | Our rating | Best for | Review |
|---|---|---|---|---|
| 1 | Bright Data Top pick | 4.7/5 | Enterprise & hard targets | Read review → |
| 2 | Oxylabs | 4.5/5 | AI & ML data collection | Read review → |
| 3 | DataImpulse | 4.0/5 | Budget-friendly scraping | Read review → |
Affiliate link. We may earn a commission on some sign-ups; it never affects our ratings.
Visit Bright DataScraping APIs and platforms
All scraping API rankings →| # | Provider | Our rating | Best for | Review |
|---|---|---|---|---|
| 1 | Bright Data Top pick | 4.7/5 | Enterprise & hard targets | Read review → |
| 2 | Apify | 4.6/5 | Ready-made scrapers | Read review → |
| 3 | Oxylabs | 4.5/5 | AI & ML data collection | Read review → |
The scraper brief
New guides, reviews and what broke on major sites, by email. No promos.