What is the Website Content Crawler used for?
It crawls an entire website (or a list of URLs) and extracts the full text, headings, metadata, and links from each page. Common uses: AI training data collection, SEO audits, competitive content analysis, site migrations, and research datasets.
Can I scrape any website with the general scraper?
The general Web Scraper handles most public websites. Heavily protected sites (financial data, ticketing platforms, heavily bot-protected e-commerce) may need specialized scrapers from the actor store with built-in anti-bot bypass.
What output formats does the Web Scraper support?
All Apify scrapers export to 7 formats: Excel (.xlsx), CSV, JSON, XML, HTML Table, RSS, and JSONL. Data can also be delivered via API, webhook, or pushed directly to cloud storage.