shield_lock
JinDaGe
← Back to skills
extension
Category: OtherAPI key required

Web Watcher Pro

Web Change Monitor — Generic webpage monitoring tool. Configure URL list → Skill checks for changes at set frequency → Feishu push notifications. Not tied to...

personAuthor: billjamno58hubclawhub
open_in_newRepositorydownloadDownload package

Web Watcher Pro

Configure any URL → Skill checks for changes at set frequency → Feishu notification.

Fully generic tool, not tied to any platform. Use cases: competitor new product alerts, price monitoring, inventory tracking, content change detection, forum thread monitoring.

Quick Start

Add a Monitored URL

User: Monitor this page: https://example.com/product/12345

Skill:

  1. Fetches page, computes content hash
  2. Asks for detection mode and frequency (or uses defaults)
  3. Saves monitoring task, begins checking

Check Status

User: Show my monitored URLs
User: Which URLs have changed?

Remove Monitor

User: Remove monitoring for https://example.com/product/12345

Detection Modes

| Mode | Description | Use Case | |------|-------------|----------| | hash | MD5 hash of full HTML, triggers on any change | General, any page | | keyword | Triggers when keyword appears/disappears | Inventory, price, specific content | | selector | CSS selector extracts specific DOM elements for comparison | List pages (product listings, search results) | | regex | Regex-defined trigger condition | Complex pattern matching |

Examples

User: Monitor this page, alert me when price drops below 99
[URL]

User: Use keyword mode, alert when product name contains "New Arrival"
[URL]

Tiered Features

| Feature | FREE | PRO | |---------|:----:|:---:| | Monitored URLs | 3 | Unlimited | | Check frequency | Every 24h | Every 1h | | Detection mode | Hash only | Hash + Keyword + Selector + Regex | | Change history | — | 30 days | | Feishu push | — | Yes | | Price | Free | $0.01/call |

Detection Modes Detail

Hash Mode

MD5 hash of full page HTML. Triggers on any content change.

Keyword Mode

Monitors for keyword appearance/disappearance. Case-insensitive.

Selector Mode

CSS selector extracts specific DOM elements. Compares extracted text between checks.

Regex Mode

Regex pattern matched against HTML. Triggers on pattern match change.

Change History

User: What pages have changed recently?
User: Show change history for https://xxx.com

Returns: change timestamp, change summary, time since last change.

Core Script

See scripts/monitor.py for full implementation:

from scripts.monitor import WebMonitor

monitor = WebMonitor(tier="pro")

monitor.add_task(
    url="https://example.com/product/123",
    name="Product A Monitor",
    mode="hash",
    frequency="6h",
)

monitor.check_all()  # Triggers Feishu push on changes
monitor.list_tasks()
monitor.remove_task(url="https://example.com/product/123")

Technical Implementation

  • Fetching: Playwright (headless) with random UA and anti-detection delays
  • Detection: MD5 hash / keyword match / CSS selector / regex
  • Storage: SQLite at /tmp/web-watcher-pro/history.db
  • Push: Feishu IM notifications with customizable templates
  • Anti-ban: Request intervals + random delays + 3x auto-retry

Security Notes

  • SSRF Protection: fetch_page() validates all URLs before sending to Playwright. Blocks: non-HTTP(S) schemes (file://, ftp://, data:, javascript:, etc.), localhost, 127.0.0.1, private IP ranges (10.x.x.x, 172.16-31.x.x, 192.168.x.x), link-local (169.254.x.x including AWS metadata 169.254.169.254), and IPv6 localhost. Unsafe URLs return None instead of triggering a network request.
  • Subprocess execution: Uses node -e subprocess for Playwright browser automation (anti-detection scraping). Node.js required. Timeout: 30s. Subprocess uses list form (not shell=True), eliminating command injection risk.
  • Data storage: Uses /tmp/web-watcher-pro/ for SQLite DB and config (no home directory write).
  • Billing data: FEISHU_USER_ID transmitted to skillpay.me/api/v1/billing for per-call charging.

Billing

  • Billing via skillpay.me/api/v1/billing/charge
  • User data transmitted to SkillPay for billing identification
  • $0.01 USD per check call (PRO tier)

Required Environment Variables

| Variable | Description | |----------|-------------| | FEISHU_USER_ID | User open_id for billing | | SKILL_BILLING_API_KEY | SkillPay Builder API Key | | SKILL_BILLING_SKILL_ID | SkillPay Skill ID (default: web-watcher-pro) |

Common Errors

| Error | Cause | Solution | |-------|-------|----------| | Failed to fetch page | Page blocked or unavailable | Check URL accessibility | | Invalid mode | Unsupported detection mode | Use: hash, keyword, selector, regex | | TASK_LIMIT_EXCEEDED | URL count exceeds tier limit | Upgrade or remove existing URLs |