Live-Data Pipeline Cookbook: MCP + n8n + LangChain (2026)

AI agents are only as good as the data they can reach. A model answering from its training cutoff will confidently describe last year’s prices, last month’s news cycle, and a documentation page that was rewritten twice since. The fix is not a bigger model — it is a pipeline that keeps fresh web data flowing into the places your agent actually reads from. This cookbook builds exactly that: Apify scrapers collect on a schedule, n8n orchestrates the run-to-index flow, a vector store holds the embeddings, LangChain serves retrieval answers, and an MCP hookup lets the agent order a brand-new scrape whenever the index is not fresh enough.

One honesty note before we start, carried over from every tutorial in this program: the wiring below is an illustrative sketch assembled from each tool’s documented pieces — the actors’ published input schemas, n8n’s built-in node types, LangChain’s documented loader and chain APIs, and our tested Apify MCP server walkthrough. It shows you the real shape of a working system, but node names and SDK imports drift between versions, so build it in staging first and adapt to whatever your installed versions show. Nothing here is a measured benchmark, and every cost figure is quoted from the actor store pages as published — verify on the pricing tab before you budget.

The Pipeline in Words: Scraper → Webhook → Vector Store → Agent

Every live-data system, from a weekend news bot to an enterprise research agent, is the same five-stage flow. Learn the stages and their handoffs and you can debug any pipeline by asking which handoff broke.

Stage 1: Scraper. One or more Apify actors run on a schedule and produce structured records. This cookbook uses three, each covering a different data shape: the Website Content Crawler for clean page text in Markdown (your docs, competitors’ changelogs, knowledge-base sources), the Google News Scraper for headlines with publishers and timestamps (media monitoring, topic tracking), and the general Web Scraper for anything else on the public web that needs a real browser. All three accept the same two documented inputs — startUrls and maxItems — so the rest of the pipeline treats them interchangeably.

Stage 2: Webhook. When a run finishes, Apify pushes the result to an HTTP endpoint instead of making you poll for it. Polling works but wastes cycles and adds latency; a webhook means the pipeline wakes up exactly when there is new data, carrying the dataset ID of the finished run. This push handoff is the heartbeat of the whole system — everything downstream reacts to it.

Stage 3: Normalize. Raw actor output is never index-ready. Records arrive with different field names per actor, HTML remnants, duplicates across runs, and timestamps in assorted shapes. A small normalize step maps every record to one canonical document — URL, title, text, source actor, crawled timestamp — and dedupes on the canonical URL so re-crawling a page updates it instead of doubling it. This is the least glamorous stage and the one that determines whether your agent quotes clean facts or garbled fragments.

Stage 4: Vector store. Each canonical document is split into chunks, embedded, and upserted into a vector database such as Qdrant or Pinecone, keyed by canonical URL. The store is the agent’s long-term memory of the web you care about: every scheduled run refreshes it, and every answer your agent gives is grounded in whatever the last refresh captured.

Stage 5: Agent. LangChain serves everyday questions from the index through a retrieval chain, and the MCP hookup gives the agent a second path — triggering a fresh actor run on demand when the question needs data newer than the last refresh. Scheduled freshness for the common case, on-demand freshness for the exceptions. That two-path design is the entire cookbook in one sentence.

Data flows in one direction — scraper fires, webhook wakes the flow, records normalize, vectors upsert, agent answers — and each stage is independently testable. Run the actor by hand and inspect the dataset, POST a saved dataset to your webhook to test normalization, query the vector store directly to test indexing, and ask the retriever a question with a known answer before you ever wire the agent loop.

The n8n Workflow, Node by Node

n8n is the visible glue: a canvas where each stage above is a node and the handoffs are wires you can inspect. Below is the workflow in node order, using only built-in n8n node types that exist in any standard installation — a Schedule Trigger, HTTP Request, Webhook, Code, IF, and the vector-store and embeddings nodes from n8n’s AI tooling. Treat node labels as illustrative and match them to your n8n version’s palette.

Node 1 — Schedule Trigger. Owns the clock. Set one trigger per cadence you need: daily for news topics, weekly for slow-moving docs. This node carries no data; it simply wakes the workflow on rhythm.

Node 2 — HTTP Request (start the actor run). Calls the Apify API to launch a run, using only the two documented input fields. One node per actor, same shape each time:

{
  "method": "POST",
  "url": "https://api.apify.com/v2/acts/datascoutapi~website-content-crawler-pro/runs?token=YOUR_API_TOKEN",
  "body": {
    "startUrls": [{ "url": "https://example.com/docs" }],
    "maxItems": 50
  }
}

Swap the actor path segment for lhotanova~google-news-scraper or apify~web-scraper and the node is identical otherwise — that interchangeability is why the pipeline standardizes on startUrls plus maxItems and nothing else.

Node 3 — Webhook Trigger (receive the finished dataset). Configure the actor run (or a follow-up HTTP Request that registers it) to POST its completion payload to this node’s URL. The payload carries the dataset ID. Splitting “start” and “receive” into two nodes matters: actor runs take minutes, and no workflow execution should sit idle waiting — the schedule fires and finishes, and a brand-new execution starts when the webhook lands.

Node 4 — HTTP Request (fetch dataset items). Reads {{$json.defaultDatasetId}} (n8n’s expression syntax referencing the webhook payload) and GETs the items from the Apify API. Keep maxItems from the run input in mind here: it caps spend per run, so a misconfigured URL list can never balloon into a giant crawl.

Node 5 — Code (normalize to canonical documents). A short JavaScript step mapping each actor’s fields onto { url, title, text, source, crawledAt } and dropping records with empty text. Per-actor field quirks live here and only here — one if branch per source actor — so adding a fourth actor later means editing this node, not rewiring the workflow.

Node 6 — IF (quality gate). Routes empty or failed batches to a notification (email, Slack) and healthy batches onward. This is the node everyone skips and everyone later wishes they had: it is what tells you a site changed its layout at 3 a.m. instead of letting silence poison the index for a week.

Node 7 — Embeddings + Vector Store (upsert). The surviving documents flow into an embeddings sub-node (your provider’s embedding model) and then a Qdrant or Pinecone Vector Store node in upsert mode, keyed by canonical URL so re-crawls update rather than duplicate. Collection or index name per data shape — one for docs, one for news — keeps retrieval precision high because the retriever never has to sift news snippets when answering docs questions.

If n8n feels like overkill, it might be: a single actor on a fixed rhythm can use the actor’s own Schedules tab plus a direct webhook to a small script that normalizes and upserts. Choose the actor-native path for one source and one rhythm; choose n8n the moment you have several actors, branching logic, or companion steps like notifications and database writes. Our RAG pipeline guide covers the ingestion and chunking patterns this workflow plugs into, in more depth.

LangChain Retriever Wiring

With vectors landing on schedule, LangChain turns the store into answers. The sketch below is Python using LangChain’s documented community pieces — ApifyDatasetLoader for ad-hoc loads, RecursiveCharacterTextSplitter for chunking, an embeddings model, a Qdrant vector store, and a retrieval chain. Illustrative, not a tested script: pin these imports to your installed langchain and langchain-community versions and expect names to have shifted.

from langchain_community.document_loaders import ApifyDatasetLoader
from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain_openai import OpenAIEmbeddings
from langchain_qdrant import QdrantVectorStore
from langchain.chains import create_retrieval_chain
from langchain.chains.combine_documents import create_stuff_documents_chain
from langchain_core.prompts import ChatPromptTemplate

# 1. Ad-hoc load: pull one finished run straight into documents.
#    Uses only the dataset ID your webhook already delivered.
loader = ApifyDatasetLoader(
    dataset_id="YOUR_DATASET_ID",
    dataset_mapping_function=lambda item: item.get("text", ""),
)
docs = loader.load()

# 2. Chunk for retrieval: overlap keeps answers from splitting mid-fact.
splitter = RecursiveCharacterTextSplitter(chunk_size=1000, chunk_overlap=150)
chunks = splitter.split_documents(docs)

# 3. Index: same Qdrant collection your n8n workflow upserts into,
#    so scheduled runs and ad-hoc loads share one memory.
vectorstore = QdrantVectorStore.from_documents(
    chunks,
    OpenAIEmbeddings(),
    collection_name="docs-index",
)
retriever = vectorstore.as_retriever(search_kwargs={"k": 5})

# 4. Answer: retrieval chain with a cite-your-sources system prompt.
system = (
    "Answer from the retrieved context only. "
    "Say when the context is insufficient."
)
prompt = ChatPromptTemplate.from_messages([("system", system), ("human", "{input}")])
chain = create_retrieval_chain(retriever, create_stuff_documents_chain(llm, prompt))
answer = chain.invoke({"input": "What changed in the docs this week?"})

Three design choices do most of the work. First, one collection per data shape (docs vs. news), mirroring the n8n upsert step, so the retriever’s top-k is never diluted by the wrong corpus. Second, metadata on every chunk — source URL, actor, crawled timestamp — so answers can cite where each fact came from and how old it is. Third, the “say when insufficient” instruction, which converts a silent knowledge gap into a visible signal — and that signal is exactly what triggers the MCP path in the next section.

MCP Hookup: Let the Agent Order Fresh Scrapes

Scheduled indexing covers every question whose answer tolerates the refresh lag. But sooner or later the agent faces a question it cannot answer from the index: breaking news since this morning’s pull, a page that changed an hour ago, a source nobody scheduled. That is when the agent should stop retrieving and start scraping — and the clean way to grant that power is Apify’s MCP server.

The full setup takes under ten minutes and is already documented step by step in our Apify MCP server tutorial: point an MCP-compatible client (Claude Desktop, Cursor, or any agent built on LangChain or LlamaIndex MCP adapters) at mcp.apify.com, authenticate once with your Apify API token, and the agent gains the entire actor marketplace as callable tools. Follow that guide first, then come back — what follows assumes the connection exists.

The cookbook contribution is the decision rule you bake into the agent’s prompt, and it is deliberately boring:

1. Answer from the LangChain retriever first, citing source URLs and their crawled timestamps.
2. If the retrieved context is insufficient or older than the question allows, call the matching actor tool with a tight startUrls + maxItems input — the same two fields the scheduled pipeline uses.
3. Summarize the fresh result for the user, and (optionally) POST it back to the n8n webhook so the next identical question hits the warm index instead of paying for another run.

Step 3 is the detail that closes the loop: on-demand scrapes become scheduled knowledge, so the system gets cheaper and faster the more it is used. And because the agent’s tool inputs use the identical two-field shape as the n8n starter nodes, the security review is simple — cap maxItems in the tool description, restrict startUrls to approved domains, and no prompt-injection surprise can turn a question into a thousand-page crawl.

Refresh Strategy and Honest Costs

A pipeline without a refresh policy is either stale or burning money. Set cadence by decision rhythm, not by enthusiasm: daily for news topics and anything where yesterday’s answer misleads (the Google News Scraper earns its keep here), weekly for documentation and evergreen sources (the Website Content Crawler’s territory), and per-question on-demand via MCP for everything else. The general Web Scraper fills gaps in any cadence when a new source appears. Record the cadence next to the data — which actor, which URLs, which date — because six months from now a strange answer is only explainable if you know exactly what was measured and when.

Costs stay honest when each stage bills separately and you quote published figures, not guesses. On the scraping side, the actors’ store pages publish their models: the Website Content Crawler follows a pay-per-result model starting at $2.97 per 1,000 results; the general Web Scraper’s actor itself is free and you pay only platform compute usage (around $0.04 per compute unit, covered by Apify’s $5 monthly credit); the Google News Scraper lists a monthly rental from $20 plus usage-based costs. Those are vendor-published figures at time of writing — confirm each on the store pricing tab, where the current numbers live, and read our Apify pricing guide for how compute units behave at scale. Your two cost levers are maxItems discipline and cadence: cap every run at the volume you will actually query, and run daily only where daily answers change decisions.

Embeddings and vector storage bill through their own providers on top — no figures quoted here because they depend on your model choice, chunk volume, and retention window, all of which this pipeline lets you measure exactly (count your upserts for a week, then price them). Start with one docs source on a weekly rhythm plus one news topic daily, watch a month of bills alongside a month of answers, and widen the crawl only when the answers prove they drive decisions. A pipeline that runs untouched for weeks on a $5 credit while answering real questions is infrastructure; a pipeline indexing the whole web that nobody queries is a hobby with a meter running.

Related Resources

• Apify MCP server tutorial
• Building RAG pipelines on web data
• Complete web scraping guide 2026
• Apify pricing guide 2026
• Browse ready-made scrapers