Scrapers Context

Overview

Web scraping infrastructure using Scrapy. Handles data acquisition from city councils (Legistar) and municipal codes (Municode).

Active Files

Spiders

• backend/affordabot_scraper/affordabot_scraper/spiders/sanjose_meetings.py - Legistar meeting scraper
• backend/affordabot_scraper/affordabot_scraper/spiders/sanjose_municode.py - Municode scraper

Configuration

• backend/affordabot_scraper/affordabot_scraper/pipelines.py - Item pipelines (DB storage)
• backend/affordabot_scraper/affordabot_scraper/settings.py - Scrapy settings

Verification Scripts

• scripts/verify_raw_scrapes.py - Check raw_scrapes table
• scripts/verify_municode_discovery.py - Verify discovery logic

Usage

Use this skill when modifying scrapers, adding new jurisdictions, or debugging ingestion.