Sitemap Generator

Crawl any website and produce a standards-compliant XML sitemap ready for search engine submission.

Quick Start

python3 scripts/sitemap_gen.py https://example.com

Output: sitemap.xml in the current directory.

Commands

# Basic — crawl and write sitemap.xml
python3 scripts/sitemap_gen.py https://example.com

# Custom output path
python3 scripts/sitemap_gen.py https://example.com -o /tmp/sitemap.xml

# Limit crawl scope
python3 scripts/sitemap_gen.py https://example.com --max-pages 500 --max-depth 3

# Polite crawling with delay
python3 scripts/sitemap_gen.py https://example.com --delay 1.0

# Set SEO hints
python3 scripts/sitemap_gen.py https://example.com --changefreq daily --priority 0.8

# Verbose progress
python3 scripts/sitemap_gen.py https://example.com -v

# Pipe to stdout
python3 scripts/sitemap_gen.py https://example.com -o -

Options

| Flag | Default | Description | |------|---------|-------------| | --output, -o | sitemap.xml | Output file path (use - for stdout) | | --max-pages | 200 | Maximum pages to crawl | | --max-depth | 5 | Maximum link depth from start URL | | --delay | 0.2 | Seconds between requests | | --timeout | 10 | Request timeout in seconds | | --changefreq | weekly | Sitemap changefreq hint | | --priority | 0.5 | Sitemap priority hint (0.0–1.0) | | --verbose, -v | off | Print crawl progress to stderr |

Dependencies

pip install requests beautifulsoup4

Notes

• Only crawls same-domain pages (no external links)
• Skips binary files (images, CSS, JS, PDFs, fonts)
• Respects the delay setting to avoid overwhelming servers
• Output conforms to the sitemaps.org 0.9 protocol