XCrawl Map

Overview

This skill uses XCrawl Map API to discover URLs for a site. Default behavior is raw passthrough: return upstream API response bodies as-is.

Required Local Config

Before using this skill, the user must create a local config file and write XCRAWL_API_KEY into it.

Path: ~/.xcrawl/config.json

{
  "XCRAWL_API_KEY": "<your_api_key>"
}

Read API key from local config file only. Do not require global environment variables.

Credits and Account Setup

Using XCrawl APIs consumes credits. If the user does not have an account or available credits, guide them to register at https://dash.xcrawl.com/. After registration, they can activate the free 1000 credits plan before running requests.

Tool Permission Policy

Request runtime permissions for curl and node only. Do not request Python, shell helper scripts, or other runtime permissions.

API Surface

• Start map task: POST /v1/map
• Base URL: https://run.xcrawl.com
• Required header: Authorization: Bearer <XCRAWL_API_KEY>

Usage Examples

cURL

API_KEY="$(node -e "const fs=require('fs');const p=process.env.HOME+'/.xcrawl/config.json';const k=JSON.parse(fs.readFileSync(p,'utf8')).XCRAWL_API_KEY||'';process.stdout.write(k)")"

curl -sS -X POST "https://run.xcrawl.com/v1/map" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer ${API_KEY}" \
  -d '{"url":"https://example.com","filter":"/docs/.*","limit":2000,"include_subdomains":true,"ignore_query_parameters":false}'

Node

node -e '
const fs=require("fs");
const apiKey=JSON.parse(fs.readFileSync(process.env.HOME+"/.xcrawl/config.json","utf8")).XCRAWL_API_KEY;
const body={url:"https://example.com",filter:"/docs/.*",limit:3000,include_subdomains:true,ignore_query_parameters:false};
fetch("https://run.xcrawl.com/v1/map",{
  method:"POST",
  headers:{"Content-Type":"application/json",Authorization:`Bearer ${apiKey}`},
  body:JSON.stringify(body)
}).then(async r=>{console.log(await r.text());});
'

Request Parameters

Request endpoint and headers

• Endpoint: POST https://run.xcrawl.com/v1/map
• Headers:
• Content-Type: application/json
• Authorization: Bearer <api_key>

Request body: top-level fields

| Field | Type | Required | Default | Description | |---|---|---:|---|---| | url | string | Yes | - | Site entry URL | | filter | string | No | - | Regex filter for URLs | | limit | integer | No | 5000 | Max URLs (up to 100000) | | include_subdomains | boolean | No | true | Include subdomains | | ignore_query_parameters | boolean | No | true | Ignore URLs with query parameters |

Response Parameters

| Field | Type | Description | |---|---|---| | map_id | string | Task ID | | endpoint | string | Always map | | version | string | Version | | status | string | completed | | url | string | Entry URL | | data | object | URL list data | | started_at | string | Start time (ISO 8601) | | ended_at | string | End time (ISO 8601) | | total_credits_used | integer | Total credits used |

data fields:

• links: URL list
• total_links: URL count
• credits_used: credits used
• credits_detail: credit breakdown

Workflow

1. Restate mapping objective.

• Discovery only, selective crawl planning, or structure analysis.

2. Build and execute POST /v1/map.

• Keep filters explicit and reproducible.

3. Return raw API response directly.

• Do not synthesize URL-family summaries unless requested.

Output Contract

Return:

• Endpoint used (POST /v1/map)
• request_payload used for the request
• Raw response body from map call
• Error details when request fails

Do not generate summaries unless the user explicitly requests a summary.

Guardrails

• Do not claim full site coverage if limit is reached.
• Do not mix inferred URLs with returned URLs.
• Do not hardcode provider-specific tool schemas in core logic.