Python Web Scraping Tutorial: Requests → Playwright → Apify API (2026)

Most Python scraping tutorials teach you exactly one tool and leave you stranded the moment the target site changes. A static product list works beautifully with requests — until the site moves its catalog behind JavaScript rendering. Playwright solves that — until you need the job to run every night, survive blocks, and deliver clean data to a teammate who does not run Python.

This tutorial teaches the full ladder instead: three levels, each building on the last. Level 1 scrapes static pages with requests and BeautifulSoup. Level 2 handles JavaScript-rendered pages with Playwright. Level 3 moves the working logic to production on the Apify API, where scheduling, proxies, and storage are managed for you. By the end you will know not just how each level works, but exactly when to graduate from one to the next.

All you need is Python 3.10 or newer and pip. Install the libraries as you go — each section lists its own one-line install command.

Level 1: Static Pages with Requests and BeautifulSoup

Most pages that matter for scraping — blog indexes, documentation, simple product listings, sitemaps — are static: the server returns the full HTML in a single response. For these, plain HTTP plus an HTML parser is the fastest, cheapest, and most debuggable stack in existence.

Install the two libraries:

pip install requests beautifulsoup4

The pattern has four steps: fetch with a polite User-Agent and a timeout, fail loudly on HTTP errors, parse the HTML, and select the elements you want with CSS selectors.

import requests
from bs4 import BeautifulSoup

HEADERS = {"User-Agent": "Mozilla/5.0 (compatible; ParseFlowTutorial/1.0)"}

response = requests.get("https://example.com/products", headers=HEADERS, timeout=15)
response.raise_for_status()

soup = BeautifulSoup(response.text, "html.parser")
for card in soup.select("div.product-card"):
    title = card.select_one("h2.title").get_text(strip=True)
    price = card.select_one("span.price").get_text(strip=True)
    link = card.select_one("a").get("href")
    print(title, price, link)

Why this works so well: requests.get performs the HTTP request with your headers and gives up after 15 seconds instead of hanging forever. raise_for_status converts 4xx and 5xx responses into exceptions so silent bad data never flows downstream. BeautifulSoup(response.text, "html.parser") builds a searchable tree from the HTML using Python’s built-in parser — no extra system dependencies. Then select and select_one accept the same CSS selectors you already know from stylesheets, get_text(strip=True) returns clean text without surrounding whitespace, and get("href") reads an attribute.

Three habits separate working Level 1 scrapers from brittle ones. First, always set a descriptive User-Agent — it identifies your script in server logs and is simply good citizenship. Second, pace your requests: a short sleep between pages keeps you far below rate limits on small jobs. Third, save one sample HTML file to disk while developing, and write your selectors against the saved copy. Re-fetching the live page on every trial run is slow and risks an accidental block before your code even works.

Level 1 hits its ceiling the day the HTML stops containing the data. If response.text has empty <div id="app"></div> placeholders where products should be, the content is rendered by JavaScript after load — and no HTTP fetcher can see it. That is the signal for Level 2.

Level 2: Dynamic Pages with Playwright

JavaScript-rendered sites — React storefronts, infinite-scroll feeds, dashboards, anything behind interactive filters — build their content in the browser after the initial HTML arrives. To scrape them you need a real browser, and Playwright is the best way to drive one from Python: one API controls Chromium, Firefox, and WebKit, with automatic waiting that eliminates most of the flakiness that plagued older browser automation.

Install it with the browser binary:

pip install playwright && playwright install chromium

The pattern mirrors Level 1, with a browser in the middle: launch Chromium headless, open a page, navigate, wait for the content selector to appear, then grab the rendered HTML and parse it with the BeautifulSoup skills you already have.

from playwright.sync_api import sync_playwright
from bs4 import BeautifulSoup

with sync_playwright() as p:
    browser = p.chromium.launch(headless=True)
    page = browser.new_page(user_agent="Mozilla/5.0 (compatible; ParseFlowTutorial/1.0)")
    page.goto("https://example.com/dynamic-list", wait_until="domcontentloaded")
    page.wait_for_selector("div.product-card")
    html = page.content()
    browser.close()

soup = BeautifulSoup(html, "html.parser")
for card in soup.select("div.product-card"):
    title = card.select_one("h2.title").get_text(strip=True)
    price = card.select_one("span.price").get_text(strip=True)
    print(title, price)

Three details do the heavy lifting here. wait_until="domcontentloaded" tells goto to return once the page skeleton is parsed rather than waiting for every image and tracker, which makes runs noticeably faster. wait_for_selector then pauses until at least one result card actually exists in the DOM — this single line replaces the fragile fixed sleeps that make naive browser scripts flaky. And page.content() returns the fully rendered HTML after JavaScript has run, so the exact same selectors from Level 1 keep working.

Playwright also unlocks interactions that HTTP can never do: clicking “load more” buttons in a loop, filling search boxes, scrolling infinite feeds until new cards stop appearing, and capturing network responses directly. But notice what Level 2 does not solve: the browser still runs on your machine, you still manage blocking and retries yourself, nothing runs while your laptop sleeps, and handing the script to a colleague means handing them your setup instructions too. When any of that starts hurting, graduate to Level 3.

Level 3: Production Scraping with the Apify API

Level 3 keeps your Python but moves execution to managed infrastructure. Instead of maintaining browsers, proxies, queues, and cron jobs, you call a maintained actor — a prebuilt, hosted scraper — through the Apify API and read the results back as structured data. Your local script shrinks to a thin client: start a run, wait, fetch the dataset.

Install the official client:

pip install apify-client

You can find your API token in the Apify Console under Settings → Integrations. The example below runs the Web Scraper — the general-purpose actor that renders pages in a real browser, mirroring what your Playwright script does — against a start URL, then iterates over every extracted record.

from apify_client import ApifyClient

client = ApifyClient("YOUR_APIFY_API_TOKEN")

run_input = {
    "startUrls": [{"url": "https://example.com/products"}],
}
run = client.actor("apify/web-scraper").call(run_input=run_input)

dataset = client.dataset(run["defaultDatasetId"])
for item in dataset.iterate_items():
    print(item.get("url"), item.get("title"))

client.actor("apify/web-scraper") addresses the actor by its store ID, and .call(run_input=run_input) starts the run and waits for it to finish — use .start() instead when you want to poll or parallelize from your own code. run["defaultDatasetId"] points at the run’s result storage, and iterate_items() streams every record as a Python dictionary, so pagination and downloads are handled for you. The same fields you selected with CSS in Levels 1 and 2 become the actor’s extraction configuration in its input — the mental model transfers directly.

If your goal is clean page text for AI workloads rather than custom per-site fields, swap in the Website Content Crawler, which returns Markdown or plain text ready for RAG pipelines. Either way, compare the actors’ store pages — including their input schemas and sample outputs — with our detail pages for the Web Scraper and the Website Content Crawler before committing to one.

Three production features come free with the move and are worth understanding. Scheduling turns a one-off script into a pipeline: in the Console, a schedule re-runs your actor’s saved input on a cron timetable, so Monday-morning datasets are simply there. Proxy and fingerprint management is built into the platform — residential rotation, browser-like TLS, and CAPTCHA handling that would be weeks of work to replicate locally. And structured storage means every run lands in a dataset you can export as Excel, CSV, or JSON, or push downstream through webhooks and integrations without touching your script.

When to Graduate from One Level to the Next

Use this decision guide honestly and you will avoid both extremes — over-engineering a one-page scrape and under-engineering a business pipeline.

Stay at Level 1 when the data is in the raw HTML, the job runs occasionally, and a few hundred pages is the whole universe. A weekly price check across twenty product pages is a perfect Level 1 job forever.

Graduate to Level 2 when the HTML arrives empty and the content renders in JavaScript, or when you need to interact with the page — search, filter, paginate, scroll. The tell is mechanical: response.text lacks the data but the browser shows it. Expect Level 2 scripts to run ten to fifty times slower per page than Level 1, which is fine for hundreds of pages and painful for hundreds of thousands.

Graduate to Level 3 when any two of these are true: the job must run on a schedule without you, blocks and CAPTCHAs are eating your mornings, the target list exceeds what your machine can chew through, or someone else depends on the data. That is the moment managed actors stop being a convenience and start being cheaper than your time — the free-tier credit covers genuine trial runs, so validate output quality before scaling.

One more rule of thumb: never reimplement at a lower level what already exists as a maintained actor. Check the actor store before writing a Level 2 script for a popular site — a dedicated scraper with handled logins, pagination, and output schemas almost certainly beats your weekend project.

Next Steps

You now have the complete ladder: fetch and parse static pages, render dynamic ones, and run production crawls through an API. Two guides take the natural next steps from here.

First, production scraping lives or dies on avoiding blocks. Read Web Scraping Without Getting Blocked to learn how sites detect bots, when residential proxies actually matter, how to pace requests with exponential backoff, and how managed actors absorb most of that complexity for you.

Second, put the ladder to work on a real project. The SERP Scraping Guide walks through a complete build — configuring a search actor, parsing its output in Python, and storing results for tracking — which exercises every level of this tutorial against live search result pages.

Pick one small target site this week, climb all three levels against it, and notice where each level strains. That felt sense of the boundaries — static versus dynamic, script versus platform — is the real skill this tutorial teaches, and it transfers to every scraping project you will touch in 2026.