Smart Web Scraper

Extract structured data from web pages into clean JSON or CSV.

Quick Start

# Scrape a page, extract all text content
uv run --with beautifulsoup4 --with lxml python scripts/scraper.py extract "https://example.com"

# Extract specific elements with CSS selector
uv run --with beautifulsoup4 --with lxml python scripts/scraper.py extract "https://example.com/products" -s ".product-card"

# Auto-detect and extract tables
uv run --with beautifulsoup4 --with lxml python scripts/scraper.py tables "https://example.com/pricing"

# Extract all links from a page
uv run --with beautifulsoup4 --with lxml python scripts/scraper.py links "https://example.com"

# Extract structured data (title, meta, headings, links)
uv run --with beautifulsoup4 --with lxml python scripts/scraper.py structure "https://example.com"

# Output as JSON
uv run --with beautifulsoup4 --with lxml python scripts/scraper.py extract "https://example.com" -s ".item" -f json

# Output as CSV
uv run --with beautifulsoup4 --with lxml python scripts/scraper.py extract "https://example.com" -s "table tr" -f csv

# Save to file
uv run --with beautifulsoup4 --with lxml python scripts/scraper.py extract "https://example.com" -s ".product" -f json -o products.json

# Multi-page scrape (follow pagination)
uv run --with beautifulsoup4 --with lxml python scripts/scraper.py crawl "https://example.com/page/1" --pages 5 -s ".article"

Commands

| Command | Args | Description | |---------|------|-------------| | extract | <url> [-s selector] [-f format] [-o file] | Extract content, optionally filtered by CSS selector | | tables | <url> [-f format] [-o file] | Auto-detect and extract all HTML tables | | links | <url> [--external] [--internal] | Extract all links (href + text) | | structure | <url> | Extract page structure: title, meta, headings, images, links | | crawl | <url> --pages N [-s selector] [-f format] [-o file] | Follow pagination links, extract from multiple pages |

Output Formats

| Format | Flag | Description | |--------|------|-------------| | Text | -f text | Plain text (default) | | JSON | -f json | Structured JSON array | | CSV | -f csv | Comma-separated values | | Markdown | -f md | Markdown-formatted |

Examples

Extract product listings

uv run --with beautifulsoup4 --with lxml python scripts/scraper.py extract "https://shop.example.com" -s ".product" -f json

Output:

[
  {"text": "Widget Pro - $29.99", "tag": "div", "class": "product"},
  {"text": "Widget Max - $49.99", "tag": "div", "class": "product"}
]

Extract pricing table

uv run --with beautifulsoup4 --with lxml python scripts/scraper.py tables "https://example.com/pricing" -f csv

Get all external links

uv run --with beautifulsoup4 --with lxml python scripts/scraper.py links "https://example.com" --external

Rate Limiting

• Default: 1 request per second (respectful crawling)
• Override with --delay 0.5 (seconds between requests)
• Respects robots.txt by default (override with --ignore-robots)

Notes

• Requires beautifulsoup4 and lxml (auto-installed by uv run --with)
• Uses a standard browser User-Agent to avoid blocks
• Handles redirects, encoding detection, and error pages gracefully
• No JavaScript rendering (use for static HTML pages)