SERP Scraping Guide 2026: Google + Bing Without an API

Search results pages are where SEO battles are won and lost — yet most teams only ever see them one keyword at a time, in their own browser, in their own location. SERP scraping changes that: it collects search engine results pages (SERPs) at scale, so you can track rankings, monitor competitors’ ads, mine content ideas, and audit local visibility across hundreds of keywords and locations.

This guide walks through the whole pattern as it works in 2026: which SERP blocks actually matter, why the official APIs disappoint, how to set up the Google Search Scraper with a working input configuration, how to parse and store the output in Python, how the same thinking applies to Bing, and where the compliance boundaries sit.

What SERP Data Actually Matters: Organic, Ads, Local Pack, PAA

A modern results page is far more than ten blue links. When you scrape a SERP, these are the blocks worth capturing:

Organic results. The classic ranked list — title, destination URL, and snippet for each position. This is the foundation of rank tracking: run the same queries daily from a fixed country and language, and you get a true time series of who ranks where. Capture the position number explicitly, because result pages increasingly interleave other blocks between organic entries.

Paid ads (top, bottom, and shopping). The Google Search Scraper extracts PPC text ads alongside organic results, including ad copy and destination URLs. For competitor research this is gold: you can see which rivals bid on your money keywords, how their copy changes over time, and which landing pages they push — intelligence no official reporting API hands you in one place.

Local pack. For location-intent queries (“dentist austin”, “best sushi near me”), Google inserts a map snippet with business listings. The scraper captures these Local Pack businesses as structured entries. If you do local SEO for clients, tracking local-pack presence separately from organic rankings is essential — a business can rank organically without appearing in the pack, and vice versa.

People Also Ask (PAA). Those expandable question boxes are a content-idea machine. The scraper extracts both the PAA questions and, distinctively, the answer snippet text revealed inside them. Seed a dozen head keywords, harvest the PAA questions, and you have a brief for FAQ sections, support articles, and long-tail targeting grounded in real searcher curiosity rather than guesswork.

Supporting blocks. Depending on the query you will also encounter the Knowledge Graph panel, related searches, and image or video carousels. The actor’s sample output centers on organicResults, paidResults, and peopleAlsoAsk, which cover the large majority of SEO and PPC use cases — start there and expand once the pipeline runs.

Why Official APIs Fall Short

Google offers an official Custom Search JSON API, and it is the right tool for one job: powering a site-internal search box with Google results. For SERP intelligence, it disappoints in three ways that scrapers do not.

First, the result set is capped and filtered. The official API returns a limited, cleaned list of results per query rather than the full page a human sees. Position tracking built on it silently misses whatever falls outside its window.

Second, the layout is gone. Ads, the Local Pack, PAA boxes, and Knowledge Graph panels are precisely the blocks that make a SERP strategically interesting — and the official API simply does not return the page as rendered. You cannot monitor competitor ad copy or mine PAA questions from an endpoint that strips them out.

Third, there is no geographic truth. Rankings differ by country, language, and even neighborhood. The scraper accepts an explicit country code, language code, and device rendering mode, so a query run truly reflects what a searcher in that market sees. Reproducing that fidelity through official endpoints requires work the scraper already does for you, with automatic proxy rotation and CAPTCHA handling keeping runs stable.

The trade-off is honest: API calls are predictable and officially sanctioned, while scraped SERP data reflects reality. For rank tracking, ad monitoring, and content research — tasks defined by “what does the searcher actually see?” — reality wins.

Scraper Setup Walkthrough With google-search-scraper

Setup takes minutes and no code. The Google Search Scraper runs on Apify’s cloud: you supply keywords and targeting, it renders the result pages and returns structured data.

Step 1: open the actor and sign in. Go to the Google Search Scraper store page and sign in with a free Apify account. Trial runs fit comfortably inside the free tier credit, so you can validate the output before spending anything.

Step 2: enter your queries. Paste one search term per line into the queries field. Start with five to ten keywords that represent one theme — for example, the product category you rank for plus two competitor brand names.

Step 3: set targeting. Fill in countryCode with the two-letter code of your market (such as us) and languageCode with the results language (such as en). Keep mobileResults set to false for desktop rankings first; run a second pass with it set to true when mobile visibility matters, since mobile and desktop SERPs genuinely differ.

Step 4: set depth. maxPagesPerQuery controls how many result pages to fetch per keyword. 1 returns the top ten results and is enough for most tracking; raise it to 2 for deeper research. Going very deep slows runs and increases usage, so reserve depth for audits rather than daily tracking.

Here is a complete input configuration using only the actor’s documented input fields:

{
  "queries": "best crm software\nproject management tools\ncheap flights",
  "countryCode": "us",
  "languageCode": "en",
  "maxPagesPerQuery": 2,
  "mobileResults": false
}

Step 5: run and export. Click Start. The actor renders each query’s result pages, extracts the blocks described above, and stores them in a dataset. When the run finishes, preview the items in the Storage tab and download them as JSON for pipelines or CSV and Excel for spreadsheets and client reports. Schedule the same input daily or weekly to turn a one-off scrape into rank tracking.

Going further with news coverage. If your research also needs publisher-side data — which outlets cover a story, when, and with what framing — pair this actor with the Google News Scraper, which extracts headlines, article URLs, publishers, and timestamps for media monitoring and share-of-voice analysis.

Parsing and Storage Pattern (Python Snippet)

Raw SERP datasets are nested: each item holds one query plus arrays of organic results, paid results, and PAA entries. The field names below (searchQuery, organicResults with title, url, description, position; paidResults; peopleAlsoAsk with question and answer) match the actor’s documented sample output. Flatten each array into its own table, keyed by query, location, and run date:

import json
import sqlite3
from datetime import date

# Load one exported dataset file (JSON array of per-query items)
with open("serp_results.json", encoding="utf-8") as f:
    items = json.load(f)

run_date = date.today().isoformat()
db = sqlite3.connect("serp.db")
cur = db.cursor()
cur.execute(
    "CREATE TABLE IF NOT EXISTS organic "
    "(run_date TEXT, query TEXT, position INTEGER, title TEXT, url TEXT, snippet TEXT)"
)
cur.execute(
    "CREATE TABLE IF NOT EXISTS paid "
    "(run_date TEXT, query TEXT, title TEXT, url TEXT, snippet TEXT)"
)
cur.execute(
    "CREATE TABLE IF NOT EXISTS paa "
    "(run_date TEXT, query TEXT, question TEXT, answer TEXT)"
)

for item in items:
    query = item.get("searchQuery", "")
    for r in item.get("organicResults", []):
        cur.execute(
            "INSERT INTO organic VALUES (?, ?, ?, ?, ?, ?)",
            (run_date, query, r.get("position"), r.get("title"),
             r.get("url"), r.get("description")),
        )
    for ad in item.get("paidResults", []):
        cur.execute(
            "INSERT INTO paid VALUES (?, ?, ?, ?, ?)",
            (run_date, query, ad.get("title"), ad.get("url"),
             ad.get("description")),
        )
    for entry in item.get("peopleAlsoAsk", []):
        cur.execute(
            "INSERT INTO paa VALUES (?, ?, ?, ?)",
            (run_date, query, entry.get("question"), entry.get("answer")),
        )

db.commit()
print(f"Stored {len(items)} queries for {run_date}")

Three habits keep this dataset useful over time. First, always store the query context (query text, country, language, device mode, run date) alongside every row — a ranking without its context is uninterpretable a month later. Second, append rather than overwrite: rank tracking, ad-copy diffing, and PAA trend analysis all require history. Third, deduplicate on (query, url, run date) before analysis, since the same URL can appear in organic results and inside PAA answers for one query.

From these three tables you can answer the questions stakeholders actually ask: “are we gaining or losing top-three positions?”, “which competitors started bidding on our brand?”, and “what new questions are searchers asking in our category?”

A Note on Bing

Everything above is described for Google, because that is where search volume and the maintained actor concentrate — but the method transfers to Bing. Bing result pages share the same anatomy (organic rankings, text ads, local answers, and a “People also search for” question block), and the same pipeline applies: collect per-query result sets with fixed geo and language settings, flatten them into dated tables, and diff over time.

For Bing specifically, use a general browser-based crawler such as the Web Scraper pointed at Bing result URLs, with modest concurrency and the same per-query-per-day storage discipline. Covering both engines matters most for US desktop audiences and for any business where Bing’s older, higher-income demographic over-indexes — run the same keyword list on both and compare share of voice engine by engine.

Compliance Notes

SERP data feels public — anyone can type a query — but “visible in a browser” and “free to collect at scale without constraints” are different claims. Our legality guide covers the full picture; the SERP-specific essentials are:

1. Read the search engine’s terms before building recurring SERP pipelines, and treat an explicit prohibition plus a warning as a stop signal.
2. Never defeat technical barriers. If a search engine blocks an IP range or serves CAPTCHAs as gates, do not build circumvention around them — use the managed actor’s built-in rotation and polite rates instead, and treat persistent blocks as an answer.
3. Keep queries and outputs clean. SERP snippets occasionally contain personal data (names, faces, contact details in local results). Minimize what you store, set retention limits, and review privacy rules for every jurisdiction whose residents appear in your data.
4. Document each project. Save the queries, geo settings, run dates, and purpose alongside the dataset — the same logging habit that makes good analytics also evidences good faith.

These notes are general information, not legal advice. For anything commercial, large-scale, or involving personal data, have qualified counsel review the plan first.

Related Resources

• Is web scraping legal in 2026?
• Web scraping without getting blocked
• Browse ready-made scrapers