Firecrawl Web Scraping

Converts web pages into clean, LLM-ready markdown or structured data. Handles JavaScript rendering, anti-bot measures, PDFs, and complex sites. No output truncation.

When to Use

| Situation | Use Firecrawl | Don't use Firecrawl | |---|---|---| | Parse PDFs | scrape with parsers option | — | | Discover all URLs on a site | map then batch-scrape | — | | Scrape 5+ pages in parallel | batch-scrape | — | | Crawl entire site recursively | crawl with depth | — | | Extract structured JSON | extract with prompt | — | | Anti-bot / Cloudflare sites | scrape with "proxy":"stealth" | — | | Fetch a single HTML page | — | Use TinyFish (returns more content) | | Semantic web search | — | Use Exa search |

Firecrawl is for crawling, mapping, PDFs, and structure. TinyFish beats it for single-page HTML.

Protocol

Scrape a Single URL

scripts/firecrawl.sh scrape "<url>" [format] [options-json]

Formats: markdown (default), html, links, screenshot

Options JSON (optional 3rd arg): {"waitFor":3000,"proxy":"stealth","parsers":[{"type":"pdf","mode":"ocr"}]}

Examples:

scripts/firecrawl.sh scrape "https://docs.firecrawl.dev/introduction"
scripts/firecrawl.sh scrape "https://spa-app.com" markdown '{"waitFor":3000}'
scripts/firecrawl.sh scrape "https://arxiv.org/pdf/2301.00001" markdown '{"parsers":[{"type":"pdf"}]}'

Batch Scrape Multiple URLs

scripts/firecrawl.sh batch-scrape '<urls-json-array>' [format] [options-json]

Processes multiple URLs in parallel via job polling. Use after map to scrape discovered pages.

Examples:

scripts/firecrawl.sh batch-scrape '["https://a.com","https://b.com","https://c.com"]' markdown
scripts/firecrawl.sh batch-scrape '["https://docs.site.com/api","https://docs.site.com/guide"]' markdown '{"waitFor":2000}'

Search Web + Scrape Results

scripts/firecrawl.sh search "<query>" [limit]

Map Website URLs

scripts/firecrawl.sh map "<url>" [limit] [search]

Discovers all URLs on a site. Use as first step before batch-scrape.

Examples:

scripts/firecrawl.sh map "https://firecrawl.dev" 50
scripts/firecrawl.sh map "https://docs.firecrawl.dev" 100 "api reference"

Extract Structured JSON

scripts/firecrawl.sh extract "<url>" "<prompt>"

Example:

scripts/firecrawl.sh extract "https://firecrawl.dev" "Extract pricing tiers with name, price, and features"

Crawl Entire Site

scripts/firecrawl.sh crawl "<url>" [limit] [depth]

Recursive crawl with job polling. Returns full markdown for each page.

Key Strengths

No output truncation — Full content passes through for all commands
PDF support — Parses PDFs natively (use parsers option for OCR on scanned docs)
Batch processing — batch-scrape handles many URLs in parallel
JS rendering — waitFor option for SPAs that need client-side rendering
Anti-bot bypass — "proxy":"stealth" for Cloudflare-protected sites
Site mapping — map discovers all URLs on a domain for targeted scraping

Critical Rules

Map before batch-scrape — Use map to discover URLs, then batch-scrape the relevant ones
Scrape for single pages — Use scrape when you have one specific URL
Use TinyFish for simple HTML fetching — TinyFish returns more content for plain HTML pages
Use Firecrawl for PDFs — TinyFish cannot parse PDFs; Firecrawl handles them natively
waitFor for SPAs — Add '{"waitFor":3000}' for JS-heavy sites that load content dynamically
Extract for structure — Use extract when you need JSON, not markdown

Resources

See reference/troubleshooting.md for error handling, configuration, and common issues.