Web Scraping Without Getting Blocked: Proxies, Fingerprints & Retries (2026)

Getting blocked is the most common reason web scraping projects fail. Your script works perfectly on page one, then suddenly returns 403 errors, CAPTCHAs, or empty pages. The site has decided you are a bot — and it is surprisingly good at telling.

This guide explains exactly how sites detect scrapers and the four techniques that keep you under the radar: smart proxy use, request pacing with retries, realistic browser fingerprints, and a CAPTCHA strategy built on avoidance rather than solving. Finally, you will see how managed Apify actors handle all of this for you when you would rather not build the infrastructure yourself.

How Sites Detect Bots

Before you can avoid detection, you need to understand what the other side is looking at. Modern bot detection is layered — no single signal gets you blocked, but a combination of small anomalies adds up to a ban.

IP reputation and request rate. Every request carries your IP address. A normal human loads a few pages per minute; a scraper without pacing can fire dozens per second from a single IP. Servers track this with rate limiting: exceed the allowed requests per time window and you get HTTP 429 responses, temporary bans, or silent connection drops. IPs from known datacenters start with a worse reputation than household connections.

Headers and TLS fingerprints. Your HTTP headers (User-Agent, Accept-Language, Referer) tell a story. A User-Agent claiming to be Chrome 126 paired with header ordering from an old Python library is an instant red flag. Below HTTP, the TLS handshake itself has a fingerprint — the cipher suites and extensions your client offers differ between real browsers and scripting libraries, and detection systems compare them against known browser profiles.

JavaScript challenges and behavior. Protected sites run JavaScript that probes your environment: screen resolution, installed fonts, canvas rendering output, mouse movements, and how quickly you scroll. Headless scripts that never move a mouse, render WebGL identically on every run, or answer challenges in zero milliseconds stand out. This is why simple requests-based scrapers die on strict sites while full-browser scrapers survive.

Honeypots and pattern analysis. Some pages include invisible links no human would ever click. Follow one and you have self-identified as a bot. Over longer runs, even well-behaved scrapers get caught by pattern analysis: perfectly even intervals between requests, 24/7 activity with no sleep cycle, or thousands of sequential product IDs are all machine-like signatures.

The takeaway: blocking is a scoring system, not a single tripwire. Every technique in this guide lowers your score on one of these layers.

Residential vs Datacenter Proxies

Proxies are the foundation of block avoidance because they solve the most visible problem — too many requests from one IP. Proxy rotation means cycling through a pool of IP addresses so each individual IP stays comfortably inside the site’s rate limits.

Datacenter proxies come from cloud hosting providers. They are cheap, fast, and available in huge quantities — ideal when you need thousands of IPs for lightly protected targets like sitemaps, public directories, or small e-commerce stores. Their weakness is reputation: detection vendors maintain lists of datacenter IP ranges, so strict sites (social networks, marketplaces) challenge or block them on sight.

Residential proxies route your traffic through real household IP addresses assigned by internet providers to home users. To the target site, your requests look like ordinary human browsing from real homes. They are far more effective against aggressive bot protection, which is why scrapers for Instagram, TikTok, and similar platforms depend on them. The trade-off is cost and sometimes speed.

A practical strategy that balances both:

1. Start with datacenter IPs for discovery, sitemap crawling, and any target that does not fight back.
2. Escalate to residential IPs only for the domains that block datacenter ranges — route per-domain, not per-project.
3. Size the pool from your target rate. If a site tolerates 30 requests per minute per IP and you need 300 per minute, you need at least 10 healthy IPs in rotation, plus spares for the ones that get burned.
4. Retire burned IPs immediately. An IP that starts receiving CAPTCHAs or 403s should leave the pool for a cooldown period instead of hammering the same wall.
5. Match geography when it matters. Prices, availability, and even page content vary by country — use IPs in the same region as the audience you are researching.

Never scrape from your own office or home IP at production volume. One aggressive run can get your everyday IP address blocklisted, which is an unpleasant way to learn about rate limits.

For the Apify-specific numbers behind these trade-offs — per-GB residential pricing, per-run proxy control, and when datacenter traffic is enough — see our Best Proxies for Apify Users in 2026 guide.

Request Pacing + Exponential Backoff

Even with a large proxy pool, hammering a server at full speed is the fastest route to a ban. Polite pacing plus intelligent retries will save more scrapers than any other single technique.

Pacing rules that work:

• Add randomized delays between requests (for example, 2–5 seconds with jitter) instead of fixed intervals — fixed intervals are a machine signature.
• Respect Retry-After headers and HTTP 429 responses: they are the server telling you the exact cooldown it wants.
• Reduce concurrency per domain. Ten parallel workers against one small site looks like an attack; spread workers across domains or time.
• Schedule large crawls outside the target’s peak hours and pause overnight to mimic human activity cycles.
• Cache aggressively. Re-scraping unchanged pages wastes requests and raises your profile for zero benefit.

Retries with exponential backoff handle the transient failures that remain. The idea is simple: when a request fails with a retryable status (429, 502, 503, or a timeout), wait before trying again — and wait longer after each consecutive failure. Here is a compact Python implementation:

import random
import time
import requests

RETRYABLE = {429, 500, 502, 503, 504}

def fetch(url, max_attempts=5, base_delay=2.0, timeout=30):
    session = requests.Session()
    session.headers.update({
        "User-Agent": (
            "Mozilla/5.0 (Windows NT 10.0; Win64; x64) "
            "AppleWebKit/537.36 (KHTML, like Gecko) "
            "Chrome/126.0.0.0 Safari/537.36"
        ),
        "Accept-Language": "en-US,en;q=0.9",
        "Accept": "text/html,application/xhtml+xml,application/xml;q=0.9,*/*;q=0.8",
    })
    for attempt in range(1, max_attempts + 1):
        try:
            response = session.get(url, timeout=timeout)
        except requests.RequestException:
            response = None
        if response is not None and response.status_code == 200:
            return response.text
        status = response.status_code if response is not None else None
        if status is not None and status not in RETRYABLE:
            response.raise_for_status()  # 4xx like 403/404 won't fix themselves
        if attempt == max_attempts:
            raise RuntimeError(f"Gave up on {url} after {max_attempts} attempts")
        # Exponential backoff with jitter: 2s, 4s, 8s, 16s (±25% random)
        delay = base_delay * (2 ** (attempt - 1)) * random.uniform(0.75, 1.25)
        # Honor the server's requested cooldown when present
        if response is not None and "Retry-After" in response.headers:
            try:
                delay = max(delay, float(response.headers["Retry-After"]))
            except ValueError:
                pass
        time.sleep(delay)

Three details make this snippet production-worthy. First, it does not retry 403s or 404s — those are permanent decisions by the server, and retrying them just confirms you are a bot. Second, the jitter (random.uniform(0.75, 1.25)) breaks the rhythmic retry pattern that detectors look for. Third, it honors Retry-After, which both reduces your block score and is simply good citizenship toward the site’s operators.

Fingerprint Basics (Headers, TLS, Canvas)

Once your IPs and pacing are sane, the next detection layer is your browser fingerprint — the dozens of small signals that distinguish a real browser from a script pretending to be one.

Headers. Send a complete, consistent set: a current User-Agent, matching Accept and Accept-Language, a plausible Referer chain (visitors arrive from search or category pages, not from nowhere), and modern Sec-CH-UA client-hint headers. Rotate through a small set of realistic profiles rather than using one identity for a million requests — and never mix parts from different browsers in one profile.

TLS and HTTP/2 fingerprints. Libraries like Python requests or basic curl produce TLS handshakes that look nothing like Chrome or Firefox. If you control the client, use libraries that impersonate real browsers at the TLS layer, or — more reliably — drive a real headless browser (Playwright or Puppeteer) so the handshake is genuine by construction. Mismatched fingerprints are one of the most common reasons datacenter scrapers fail on Cloudflare-protected sites.

Canvas and rendering signals. Fingerprinting scripts ask your browser to draw hidden text or 3D scenes and hash the pixels. Real devices produce slightly different output due to GPU and font differences; identical canvas hashes across thousands of “visitors” scream automation. Full browsers with varied viewports, real font sets, and human-like interaction (scrolling, hovering, variable timing) pass these checks far more often than raw HTTP clients.

The pragmatic rule: match the tool to the target. Plain HTTP with good headers and pacing handles most blogs, docs, and small stores. Anything behind serious bot management needs a real browser with residential IPs — which is exactly the setup managed platforms provide out of the box.

CAPTCHA Strategy (Avoid > Solve)

A CAPTCHA appearing in your scrape is not a puzzle to solve — it is a symptom that your earlier stealth failed. The site already suspects you; solving the challenge just buys one more request before the next, harder check. Build your strategy around never seeing one:

1. Slow down first. CAPTCHAs are usually triggered by rate thresholds. Dropping your per-IP request rate and adding jitter eliminates the majority of them.
2. Upgrade the IP, not the solver. A residential IP with a clean reputation sails through pages where a datacenter IP gets challenged on every load.
3. Use full-browser sessions with cookies. Maintaining session state, accepting cookies, and warming up with a few ordinary page views builds trust before you hit the pages you actually need.
4. Avoid trigger actions. Logging in, rapid-fire searching, and aggressive pagination are the behaviors most likely to summon a challenge. Space them out or use official data paths where they exist.
5. Treat solving services as a last resort. Third-party solvers add per-request cost, latency, and failure modes — and they do nothing against invisible behavioral scoring that blocks you without showing any challenge at all.

If a target shows CAPTCHAs to residential IPs at human-like speeds inside a real browser, take the hint: that data source wants you to slow down dramatically or negotiate access. Pushing through with solver farms is how projects graduate from “blocked” to “IP range banned and legal letter.”

How Apify Actors Handle This for You

Everything above is infrastructure you can build — proxy pools, backoff logic, browser fleets, fingerprint management — or rent. Managed Apify actors bundle all four layers, which is why they are the fastest path from “blocked” to “data delivered” for most teams:

• Web Scraper runs your crawl in a full Chromium browser on Apify’s cloud with proxy rotation, concurrency management, and retries built in. You describe what to extract; the actor handles dynamic pages, infinite scroll, and export to Excel, CSV, JSON, XML, HTML, RSS, or JSONL. The actor itself is free — you only pay Apify platform compute usage ($0.04 per compute unit on the Free plan), covered by the $5 monthly credit.
• Instagram Scraper exists precisely because Instagram has some of the strictest anti-bot protections on the web. It ships with automatic proxy handling designed for those defenses, extracting public profiles, posts, reels, hashtags, and engagement metrics. It uses a pay-per-result model at $1.50 per 1,000 results, so roughly 3,300 posts or profiles fit inside the $5 free monthly credit.
• Tweet Scraper V2 sustains 30–80 tweets per second across keyword searches, timelines, lists, and threads by fanning runs out across Apify’s cloud — throughput that would get a self-hosted IP banned within minutes. It costs $0.40 per 1,000 tweets (minimum 50 tweets per query), and note that free-plan users are limited to 5 runs per month.

The pattern is the same in each case: the anti-blocking expertise (IP pools, pacing, fingerprints, session handling) lives inside the actor, maintained by people who update it every time the target site changes its defenses. Your code shrinks to an API call.

Checklist: Scrape Without Getting Blocked

Run through this list before every production crawl:

• Rotate IPs with a pool sized to your target rate; reserve residential proxies for strict domains.
• Pace requests with randomized delays and per-domain concurrency limits — never fixed intervals.
• Retry intelligently with exponential backoff, jitter, and Retry-After support; never retry 403s.
• Send consistent fingerprints: complete headers, real-browser TLS, varied viewports for JavaScript-heavy targets.
• Avoid CAPTCHAs by slowing down and upgrading IPs rather than buying solver services.
• Cache and deduplicate so you never pay for (or get flagged for) the same page twice.
• Monitor block signals — 403/429 rates, CAPTCHA frequency, empty responses — and auto-pause when they spike.
• Consider a managed actor when the target’s defenses cost more engineering time than the data is worth.

Blocking is not a wall; it is a negotiation. Sites set the price of their data in politeness, and scrapers that pay it — with good IPs, human pacing, honest fingerprints, and restraint — collect reliably for years. Start polite, escalate infrastructure only where needed, and hand the hardest targets to managed actors built for exactly that fight.