Scraper

Turn messy public pages into clean, reusable data.

Core Purpose

Scraper is a safe extraction skill for public, user-authorized pages. It helps the agent:

• fetch page content from a URL
• extract readable text
• strip boilerplate where possible
• save clean output locally
• prepare content for later summarization or analysis

Safety Boundaries

• Only use on public or user-authorized pages
• Do not bypass logins, paywalls, captchas, robots restrictions, or rate limits
• Do not request or store credentials
• Do not perform stealth scraping, account creation, or identity evasion
• Save outputs locally only

Runtime Requirements

• Python 3 must be available as python3
• No external packages required

Local Storage

All outputs are stored locally under:

• ~/.openclaw/workspace/memory/scraper/jobs.json
• ~/.openclaw/workspace/memory/scraper/output/

Key Workflows

• Capture a page: fetch_page.py --url "https://example.com"
• Extract readable text: extract_text.py --url "https://example.com"
• Save cleaned content: save_output.py --url "https://example.com" --title "Example"
• List prior jobs: list_jobs.py

Scripts

| Script | Purpose | |---|---| | init_storage.py | Initialize scraper storage | | fetch_page.py | Download a page with standard headers | | extract_text.py | Convert HTML into cleaned plain text | | save_output.py | Save extracted output and register a job | | list_jobs.py | Show past scraping jobs |