shield_lock
JinDaGe
← Back to skills
extension
Category: Data & AnalyticsAPI key required

Anycrawl

Web scraping, search, crawling, and site mapping via the AnyCrawl CLI. Use when the user wants to search the web, scrape a page, find URLs on a site, or bulk...

personAuthor: qthanshubclawhub
open_in_newRepositorydownloadDownload package

AnyCrawl CLI

Web scraping, search, and crawling CLI. Returns clean markdown optimized for LLM context windows. Default engine: playwright.

Run anycrawl --help or anycrawl <command> --help for full option details.

Prerequisites

Must be installed and authenticated. Run anycrawl login or set ANYCRAWL_API_KEY.

If not ready, see rules/install.md. For output handling guidelines, see rules/security.md.

Commands

  • Search - No specific URL yet. Find pages, answer questions. Use --scrape to get full page content with results.
  • Scrape - Have a URL. Extract its content directly.
  • Map - Need to locate a specific page on a site. Discover URLs, then scrape the ones you need.
  • Crawl - Need bulk content from a site or section. Use crawl directly — no need for map first.

| Need | Command | When | | --------------------------- | -------- | ---------------------------------- | | Find pages on a topic | search | No specific URL yet | | Get a page's content | scrape | Have a URL | | Find URLs within a site | map | Need to locate a specific subpage | | Bulk extract a site section | crawl | Need many pages (e.g., all /docs/) |

For detailed command reference, run anycrawl <command> --help (e.g., anycrawl search, anycrawl scrape).

Avoid redundant fetches: search --scrape already fetches full page content. Don't re-scrape those URLs. Check .anycrawl/ for existing data before fetching again.

Output & Organization

Write results to .anycrawl/ with -o. Add .anycrawl/ to .gitignore. Always quote URLs in shell commands. Never read entire output files at once — use grep, head, or incremental reads.

Documentation