Web Navigation with SeaPortal

CLI-first read-only web fetcher. Use the seaportal command. No JavaScript execution — for SPAs/blocked pages, escalate to a real browser.

Core Commands

seaportal <url>                              # Markdown + frontmatter to stdout (--save-dir <dir> to also write files)
seaportal --json <url>                       # Full Result struct as JSON
seaportal --xml <url>                        # TEI-Lite XML (teiHeader metadata + text/body content)
seaportal --snapshot <url>                   # Accessibility tree as JSON
seaportal --snapshot --format=compact <url>  # Accessibility tree as text (most token-efficient)
seaportal --snapshot --filter=interactive --format=compact <url>  # Only links/buttons/inputs
seaportal --max-tokens=2000 <url>            # Cap Markdown body size (paragraph-boundary cut, sets truncated:true)
seaportal --snapshot --max-tokens=2000 <url> # Cap snapshot tree size
seaportal --fast <url>                       # Bail early if browser is needed
seaportal --head-only <url>                  # 16 KB range fetch — metadata + canonical only, no body
seaportal --respect-robots <url>             # Consult robots.txt; refuse disallowed fetches
seaportal --rate-limit=500ms <url>           # Min interval between same-host requests
seaportal --probe-search <url>               # Override to needs-browser when search URL yields no results
seaportal --no-dedupe <url>                  # Disable repeated-block dedup
seaportal --no-prune-fallback <url>          # Disable the heuristic fallback when readability is thin
seaportal --with-links <url>                 # Add structured list of discovered <a> links to output
seaportal --with-images <url>                # Add structured list of discovered <img> entries to output
seaportal --with-tables <url>                # Add structured tables (caption/headers/rows) to output, layout tables flattened
seaportal --with-comments <url>              # Emit user-generated comments (Disqus/native) separately in result.comments (stripped from Content by default either way)
seaportal --links=text <url>                 # Markdown link retention: none|text|all|footer (default all)
seaportal --citations <url>                  # Synonym for --links=footer (back-compat)
seaportal --chunk=heading <url>              # Chunk Markdown by heading / sentence / window
seaportal --select=".main-content" <url>     # Scope extraction to a CSS subtree
seaportal --strip=".ads,.cookie-banner" <url> # Remove matching elements before extraction
seaportal --retries 5 <url>                  # Override default retry count (3)
seaportal --max-retry-wait 10s <url>         # Override single backoff cap (30s)
seaportal --retry-timeout 60s <url>          # Override total retry budget (90s)
seaportal --base-url=URL -                   # Read HTML from stdin; --base-url resolves relative links
seaportal --ua=googlebot <url>               # User-Agent preset or literal string
seaportal --proxy=http://user:pass@proxy:8080 <url>  # Route via HTTP(S) proxy
seaportal --cache=/tmp/sp-cache <url>        # Reuse fresh 200 OK responses from disk (opt-in; default TTL 24h)
seaportal --cache=/tmp/sp-cache --cache-stale-tolerance=5m <url>  # Stale-while-revalidate: serve stale within TTL+tolerance, refresh in background
seaportal --no-pdf <url>                     # Skip PDF extraction (default: extract PDF text)
seaportal --schema=path/to/schema.yaml <url> # Apply a CSS schema (JSON/YAML), populate result.schema
seaportal --query="compound interest" <url>  # BM25-rank H2/H3 sections by relevance, annotate result.rankedSections
seaportal --query=... --top-n=3 <url>        # Keep only the top-N most relevant sections in rankedSections
seaportal --query=... --top-n=3 --filter-by-query <url>  # Replace Content with concatenated top-N sections
seaportal --split-out=dir/ --split-bytes=32768 <url>  # Shard output across multiple files; print manifest (path\tindex/of\tbytes) to stdout
seaportal scrape <url> --preview             # Map a site's tree: 1 sample per URL pattern + counts (see "Scraping a whole site")
seaportal scrape <url> --recent-days 7       # Scrape only sitemap URLs modified in the last 7 days (required for big news sites)
seaportal scrape <url> --include-patterns '/blog/*' --output md  # Expand one branch into a Markdown digest
seaportal --version

Subcommands

The default verb (no subcommand) is URL extraction — seaportal <url> behaves exactly as documented above.

• seaportal sitemap <url> — fetch a sitemap.xml, recurse into nested <sitemapindex> references, decompress .gz, and print one URL per line. Flags: --json (emit JSON array of {loc,lastmod,changefreq,priority} entries), --max-urls N (default 50000), --max-depth N (default 5), --allow-internal (permit trusted private/internal hosts). Example: seaportal sitemap https://example.com/sitemap.xml --json.
• seaportal feed <url> — fetch and parse RSS 2.0, Atom 1.0, or JSON Feed 1.x into a unified {title, link, published, summary, author, guid} shape (format sniffed from the root element / first byte). Default output is one TSV line per item (published\ttitle\tlink). Flags: --json (emit JSON array), --max-items N (default 200), --allow-internal (permit trusted private/internal hosts). Example: seaportal feed https://example.com/feed.xml --json.
• seaportal scrape <url> — scrape a whole site: discover URLs (robots.txt + sitemap, or a bounded homepage crawl when there is no sitemap), cluster them into path patterns, sample within a budget, and fetch + extract each page concurrently. See Scraping a whole site below. Key flags: --preview (map the site tree cheaply — 1 sample per pattern with counts), --recent-days N (bound sitemap discovery to the last N days — required for large news sites), --max-pages N (default 50), --max-per-pattern N (default 8), --full (fetch all discovered pages, no sampling), --include-patterns / --exclude-patterns (comma-separated globs), --output json|md|directory (+ --out-dir for directory), --allow-internal.
• seaportal snapshot <url> — print the accessibility-tree snapshot for a URL (the real subcommand form of --snapshot; see Thin Markdown? Try the snapshot). Flags: --filter interactive, --format json|compact, --max-tokens N.
• seaportal mcp — run as an MCP (Model Context Protocol) server over JSON-RPC 2.0 line-delimited stdio. Exposes five tools — fetch_url, fetch_snapshot, parse_sitemap, parse_feed, scrape_site — each routing to the library entry point of the same shape. No flags; configuration flows through MCP tool arguments. See MCP integration below.
• seaportal help — usage summary including subcommands.

MCP integration

Register seaportal as an MCP server in your editor (Claude Desktop / Claude Code / Cursor) — example claude_desktop_config.json:

{
  "mcpServers": {
    "seaportal": {
      "command": "seaportal",
      "args": ["mcp"]
    }
  }
}

Tools exposed: fetch_url ({url, dedupe?, fast?, with_links?, with_images?, with_tables?, with_comments?, max_tokens?}), fetch_snapshot ({url, filter?, max_tokens?, allow_internal?}), parse_sitemap ({url, max_depth?, max_urls?, allow_internal?}), parse_feed ({url, max_items?, allow_internal?}), scrape_site ({base_url, max_pages?, max_per_pattern?, include_patterns?, exclude_patterns?, allow_internal?}). Each returns its library result as a single JSON text content block.

User-Agent presets

--ua=<name> accepts curated presets; unknown values pass through as literal UA strings. Empty (default) sends a real Chrome UA. Per-host DomainUserAgent overrides still win.

• chrome (default), safari, firefox — real browser UAs
• googlebot, bingbot, search-bot — bot UAs (may trigger reverse-DNS challenges)
• seaportal — honest self-identify for cooperative sites

Cache

--cache=<dir> is opt-in: only fresh 200 OK responses are stored, keyed by SHA-256 of URL + Accept/Accept-Language/User-Agent. --cache-ttl=<dur> controls freshness (default 24h). --no-cache bypasses reads but still writes — i.e. "force refresh". Errors / 3xx / 4xx / 5xx are never cached. Result includes cacheHit: true on a replay.

Past-TTL entries that carry ETag or Last-Modified are automatically re-validated with a conditional GET (If-None-Match / If-Modified-Since). A 304 Not Modified replays the cached body, refreshes FetchedAt, and sets cacheRevalidated: true (distinct from cacheHit, which means "served from disk with no network call"). A 200 replaces the entry; other statuses leave the cache untouched.

--cache-stale-tolerance=<dur> enables stale-while-revalidate (SWR) semantics: entries whose age falls within TTL + tolerance are served immediately from disk (with cacheStale: true) while a background goroutine fires the conditional GET to refresh the entry for the next call. Default 0 keeps the existing synchronous-revalidate behaviour. The SWR band does not require validators on the cached entry — within tolerance the body is trusted unconditionally; beyond tolerance the existing validator-gated synchronous revalidation runs. --no-cache bypasses SWR entirely. Background refresh failures are silent and leave the stale entry intact for the next attempt.

PDF extraction

application/pdf responses are extracted by default: text is pulled page-by-page via ledongthuc/pdf and flows through the same Result.Content Markdown pipeline (link retention, truncation, chunking, cache) with ExtractionMethod="pdf" and --- page N --- separators. Pass --no-pdf to restore the legacy "skipped binary content" behaviour. Image-only / scanned PDFs yield an empty extraction error (no OCR).

HTTP transport

• HTTPS connections negotiate HTTP/2 via ALPN when the server offers h2; fall back to HTTP/1.1 otherwise.
• HTTP (no TLS) always uses HTTP/1.1.
• All HTTPS connections use a Chrome 122 TLS fingerprint via utls — bypasses Cloudflare's Go-default-TLS bot detection.
• The negotiated protocol is surfaced on Result.protocol ("h2" or "http/1.1").

Proxy support

--proxy=URL routes the fetch through a proxy. http:// and https:// proxy URLs are supported with Basic auth taken from the URL userinfo (user:pass@host:port). HTTPS targets use a CONNECT tunnel; the Chrome TLS fingerprint is preserved end-to-end with the origin. socks5:// URLs work for HTTP target URLs only — HTTPS-over-SOCKS5 is a V1 limitation. Invalid proxy URLs fail fast with result.Error = "invalid proxy URL: ...".

Security defaults (local / internal URLs)

The CLI is safe by default (DefaultSecurityPolicy): it blocks targets that resolve to private/internal IPs (SSRF guard), allows only http/https, caps redirects at 10 with per-hop re-validation, and bounds the raw (50 MiB) and decompressed (200 MiB) body.

• Reading localhost, 127.0.0.1, a 192.168.*/10.* host, or any intranet URL fails by default with Error: target resolves to a private/internal IP. Add --allow-internal to permit it (you are vouching the target is trusted).
• Other knobs: --max-redirects N, --allow-domains / --deny-domains, --trusted-resolve-cidrs, --max-response-bytes, --max-decompressed-bytes.
• The MCP server applies the same safe default to fetch_url, fetch_snapshot, parse_sitemap, and parse_feed.
• Caveats: with --proxy the dial-time rebinding check is skipped (the target is still vetted before fetch and on each redirect, just not at connect time).

Per-host rate limiting

--rate-limit=DURATION enforces a minimum interval between requests to the same host. Useful primarily for library callers sharing a HostRateLimiter across calls via Options.RateLimiter; a single CLI invocation only fires one request, so the throttle has no cross-call effect by itself. Combines with --respect-robots crawl-delay (both apply; effective wait is their sum).

Chunking

--chunk=NAME[:SIZE[:OVERLAP]] populates result.chunks (off by default). Runs after --max-tokens truncation; fenced code blocks are never split.

• --chunk=heading — split at H2/H3 boundaries; pre-heading prologue is its own chunk.
• --chunk=sentence:512 — group sentences until ~512 tokens; heading inherited from nearest H2/H3 above.
• --chunk=window:2000:200 — 2000-char windows, 200-char overlap, snapped to word boundaries.

Query relevance

--query="..." scores each H2/H3-bounded Markdown section against the query with standard BM25 (k1=1.5, b=0.75) and populates result.rankedSections (descending score). Pure additive by default — Content is untouched. Combine with --top-n=N to truncate, or --filter-by-query to replace Content with the concatenated top-N sections (default top-3 when --top-n is unset). No stopwords/stemming in V1; IDF handles common words naturally.

• seaportal --json --query="formula" <url> — annotate all sections with scores.
• seaportal --json --query="formula" --top-n=3 <url> — keep only the 3 highest-scoring sections in rankedSections.
• seaportal --query="formula" --top-n=3 --filter-by-query <url> — Markdown body becomes just those 3 sections.

Schema extraction

--schema=<path> applies a declarative CSS schema (JSON or YAML, format sniffed from extension) to the raw HTML and surfaces the result as result.schema in JSON output. Three modes per field: single value (selector only), multiple values (multiple: true), nested array of objects (fields: populated). Optional attr: reads an attribute instead of text. Runs on the pre-preprocess DOM so chrome elements (nav/sidebar) are reachable. Bad selectors or load failures become warnings, never crash. Example schema:

fields:
  title: { selector: h1 }
  tags:  { selector: .tag, multiple: true }
  products:
    selector: .product
    fields:
      name:  { selector: .name }
      price: { selector: .price, attr: data-price }

Invoke: seaportal --json --schema=./schema.yaml <url>. XPath is a V2 limitation — CSS only for now.

Chaining with a browser fetcher

When a page needs JS execution, let pinchtab (or any other fetcher) render the HTML, then pipe it into seaportal for extraction:

pinchtab fetch https://example.com | seaportal --base-url https://example.com --json -

--base-url is required in stdin mode. Network-only flags (--head-only, --respect-robots, --retries) are silently no-ops with a stderr warning.

Workflow: navigating a site

1. Fetch the entry point as Markdown: seaportal <url>. Read the frontmatter — pageClass, trustworthy, needsBrowser, confidence tell you if extraction is reliable.
2. Decide next step from pageClass:
   • static / ssr / hydrated → trustworthy, keep using seaportal.
   • spa / dynamic → JS-only content, stop and escalate to a browser (pinchtab).
   • blocked → bot-protection or login wall, escalate.
3. Discover links: extract URLs from the Markdown body, OR run seaportal --snapshot --filter=interactive --format=compact <url> to see only links/buttons with their hrefs. Each entry has a stable ref (e1, e2…) and CSS selector.
4. Follow a link: take the href, resolve against the page URL if relative, and re-run seaportal <new-url>.
5. Repeat until you have what you need. Track visited URLs to avoid loops.

There is no session, no click, no form submit — every navigation is a fresh HTTP GET. To "click" a link you re-invoke seaportal on its href.

Scraping a whole site

seaportal scrape <url> handles multi-page sites in one shot: it discovers URLs (robots.txt Sitemap: directives + the conventional /sitemap.xml, recursing sitemap-indexes; or a bounded homepage crawl when there is no sitemap), groups them into path patterns like /blog/*/* , samples within a budget, then fetches and extracts every sampled page concurrently.

Preview first, then expand. Don't blind-scrape a large site — you'll fetch the wrong pages and waste budget. Map the tree first:

1. Preview the tree: seaportal scrape https://site.com --preview. This takes 1 representative sample per URL pattern and reports a per-group count (sampled 1 of 87), so you see the shape of the site — which sections exist and how big each is — for the price of a handful of fetches. --preview implies recent-only (last 7 days) and never does a full fetch.
2. Read the page groups: each group is a path pattern with a count. Pick the branch(es) you actually want (e.g. /economia/*/*/*/news/* had 112 URLs).
3. Expand the chosen branch: seaportal scrape https://site.com --include-patterns '/economia/*' --recent-days 7 --max-per-pattern 20. Use --exclude-patterns to prune noise, --full to take everything in the filtered set.

Large news / archive sites need --recent-days. Sites like repubblica.it publish a sitemap index of monthly .gz archives spanning years — hundreds of thousands of URLs. Flattening all of it is prohibitive. --recent-days N skips any child sitemap (and any <url>) whose <lastmod> is older than N days, so discovery only walks recent partitions. Without it, discovery of such a site can run for minutes. --preview sets a 7-day window by default; pass --recent-days N explicitly to widen or narrow it (undated entries are always kept).

Output: --output json (default; site, pageGroups[] with counts, pages[], summary), --output md (a readable digest — the page-group list is the site map), or --output directory --out-dir DIR (one .md per page plus result.json). Scrape is secure-by-default — to scrape a private/internal host add --allow-internal after the scrape subcommand.

Choosing output format

| Goal | Command | |---|---| | Read article / docs content | seaportal <url> (Markdown) | | Programmatic decision-making | seaportal --json <url> | | Map of page structure (cheap on tokens) | seaportal --snapshot --format=compact <url> | | Just the actionable links/buttons | seaportal --snapshot --filter=interactive --format=compact <url> | | Large page, must cap tokens | add --max-tokens=2000 (caps snapshot tree in snapshot mode, or Markdown body otherwise; truncates at the latest paragraph boundary and sets truncated: true) |

Compact snapshot rows look like:

e2 link "Docs" <a> [interactive] href=/docs
e5 heading "Welcome" <h1> level=1

Classification cheat sheet (when to escalate)

pageClass field in the frontmatter / JSON:

| Class | Action | |---|---| | static | Use seaportal — pure HTML, high confidence. | | ssr | Use seaportal — server-rendered. | | hydrated | Use seaportal — SSR + JS, usually fine. | | spa | Escalate — JS-only, seaportal will return little/empty. | | dynamic | Escalate — heavy client rendering. | | blocked | Escalate — bot protection, captcha, or login wall. |

Also escalate if needsBrowser: true or validationOk: false. Use --fast when you want seaportal to bail early on any of these instead of doing full extraction.

For programmatic routing, prefer profile.decision + profile.browserRecommended over mapping pageClass yourself: one explicit decision (static-high-confidence / static-ok / static-caution / browser-needed / blocked / unreachable / not-found / unsupported), where browserRecommended: true means a real browser is likely to help. See docs/reference/browser-discriminator.md.

Thin Markdown? Try the snapshot before escalating

If seaportal classified the page as static/ssr/hydrated (i.e. it thinks extraction succeeded) but the Markdown body looks thin — length < ~1500, no real paragraphs, mostly headings or naked links — don't escalate yet. Readability sometimes prunes link-heavy or table-heavy sections that the accessibility tree still has in full. Retry with:

seaportal --snapshot --format=compact <url>                    # whole structure
seaportal --snapshot --filter=interactive --format=compact <url>  # just links/buttons

If the snapshot returns substantial nodes, use those. If the snapshot is also thin, then escalate. Canonical situations where the fallback is worth a try: government index pages (usa.gov, gov.uk) where the link set is the content, and reference/listing pages where the prose is incidental to the structure. Don't loop — one snapshot retry is enough; if it's not there, it's not coming.

For search URLs specifically, --probe-search short-circuits CNN/DDG-style JS search shells with reason client-rendered-search — when set, seaportal flips outcome to needs-browser if a search-shaped URL returns a short body with no result-list structure.

Output side-effects

All modes print to stdout only by default. Pass --save-dir <dir> to also write the rendered Markdown and JSON to <dir>/<domain>_<timestamp>.{md,json}.

TEI-Lite XML output (--xml)

--xml emits a TEI-Lite-shaped document: <teiHeader> with title/author/published-date/language/source URL, plus <text><body> containing the Markdown converted into <head>, <list>/<item>, <code>, and <p> elements. Mutually exclusive with --json (exit 2). Scope is basic structural shaping — full TEI ODD validation, footnotes, and cross-references are out of scope.

Output splitting

--split-out=<dir> shards the rendered output into multiple files under <dir>, capped at --split-bytes (default --max-tokens × 4 or 32 KB). Prefers existing --chunk boundaries; otherwise splits on paragraph boundaries. Files are named <url-slug>-NNN.{md,json} and written atomically (.tmp + rename). Stdout receives a TSV manifest (path\tindex/of\tbytes) instead of the content body. Not supported with --xml.

What this skill is NOT

• Not a browser. No JS execution, no clicks, no form fills, no cookies/sessions.
• Not stateful. Each call is independent.
• For interactive flows, multi-step forms, or auth — use the pinchtab skill instead. A common pattern is: seaportal first, pinchtab on escalation.