Olostep — Web Data API for AI Agents

Fetch live web content via the Olostep API. Covers scraping, searching, crawling, batch processing, site mapping, AI-powered answers, and structured data extraction.

Authentication: Every request needs Authorization: Bearer $OLOSTEP_API_KEY. If the env var is missing, stop and tell the user to set it. Get a free key (500 req/month) at https://olostep.com/auth.

Base URL: https://api.olostep.com/v1

1. Scrape a Single Page

Extract content from any URL as markdown, HTML, JSON, or text. Handles JavaScript rendering and anti-bot protections automatically.

curl -sS -X POST "https://api.olostep.com/v1/scrapes" \
  -H "Authorization: Bearer $OLOSTEP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url_to_scrape": "https://example.com/page",
    "formats": ["markdown"]
  }'

Response: Content is in result.markdown_content (or result.html_content, result.text_content, result.json_content depending on requested formats).

| Parameter | Required | Default | Description | |-----------|----------|---------|-------------| | url_to_scrape | Yes | — | URL to scrape | | formats | Yes | — | Array: markdown, html, text, json, screenshot | | country | No | — | Country code for geo-targeted scraping (US, GB, IN) | | wait_before_scraping | No | 0 | Milliseconds to wait for JS rendering (0–10000) | | parser | No | — | Parser object {"id": "@olostep/google-search"} for structured JSON | | llm_extract | No | — | Object with schema for LLM-based extraction |

When to use: Single page extraction — docs, articles, product pages, profiles.

Tips:

Default to formats: ["markdown"] — most token-efficient for LLM processing
For JavaScript-heavy SPAs, set wait_before_scraping: 2000
Use parsers for structured JSON from known sites (see Parsers section)

2. Search Google

Search Google by scraping a Google URL with the @olostep/google-search parser. No separate search endpoint — search goes through /v1/scrapes.

curl -sS -X POST "https://api.olostep.com/v1/scrapes" \
  -H "Authorization: Bearer $OLOSTEP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url_to_scrape": "https://www.google.com/search?q=best+AI+coding+tools+2026&gl=us",
    "formats": ["json"],
    "parser": {"id": "@olostep/google-search"}
  }'

Response: result.json_content is a stringified JSON string. Parse it to get organic (array of {title, link, snippet}), knowledgeGraph, peopleAlsoAsk, relatedSearches.

How to build the Google URL:

Base: https://www.google.com/search?q=YOUR+QUERY
Add &gl=us for country (ISO codes: us, gb, de, in)
URL-encode the query (spaces become +)

When to use: Research, finding docs, competitive analysis, debugging errors.

3. Crawl a Website

Async crawl that discovers and scrapes pages by following links. Poll for results.

# Start crawl
curl -sS -X POST "https://api.olostep.com/v1/crawls" \
  -H "Authorization: Bearer $OLOSTEP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "start_url": "https://docs.example.com",
    "max_pages": 10
  }'

# Check status (poll until status is "completed")
curl -sS "https://api.olostep.com/v1/crawls/CRAWL_ID" \
  -H "Authorization: Bearer $OLOSTEP_API_KEY"

# Get pages (once completed)
curl -sS "https://api.olostep.com/v1/crawls/CRAWL_ID/pages?limit=10" \
  -H "Authorization: Bearer $OLOSTEP_API_KEY"

Pages return retrieve_id per page. Use /v1/retrieve?retrieve_id=ID&formats=markdown to get content.

| Parameter | Required | Default | Description | |-----------|----------|---------|-------------| | start_url | Yes | — | Starting URL | | max_pages | Yes | — | Maximum pages to crawl | | include_urls | No | ["/**"] | Glob patterns to include (["/blog/**"]) | | exclude_urls | No | — | Glob patterns to exclude (["/admin/**"]) | | max_depth | No | — | Maximum link depth from start URL |

When to use: Ingesting docs sites, blog archives, product catalogs.

4. Batch Scrape URLs

Scrape up to 10,000 URLs in one parallel batch. Async — poll for results.

curl -sS -X POST "https://api.olostep.com/v1/batches" \
  -H "Authorization: Bearer $OLOSTEP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "items": [
      {"url": "https://example.com/page1", "custom_id": "page1"},
      {"url": "https://example.com/page2", "custom_id": "page2"}
    ]
  }'

# Check status
curl -sS "https://api.olostep.com/v1/batches/BATCH_ID" \
  -H "Authorization: Bearer $OLOSTEP_API_KEY"

# Get results (once completed)
curl -sS "https://api.olostep.com/v1/batches/BATCH_ID/items?limit=20" \
  -H "Authorization: Bearer $OLOSTEP_API_KEY"

Items return retrieve_id. Use /v1/retrieve?retrieve_id=ID&formats=markdown for content.

When to use: Large-scale extraction — product pages, directories, documentation sets.

5. Map a Website

Discover all URLs on a site without scraping content. Synchronous — returns immediately.

curl -sS -X POST "https://api.olostep.com/v1/maps" \
  -H "Authorization: Bearer $OLOSTEP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "url": "https://example.com",
    "include_urls": ["/blog/**"],
    "top_n": 50
  }'

Response: urls array of discovered URLs, urls_count total.

| Parameter | Required | Default | Description | |-----------|----------|---------|-------------| | url | Yes | — | Website to map | | search_query | No | — | Sort URLs by relevance | | top_n | No | — | Limit number of URLs | | include_urls | No | — | Glob patterns to include | | exclude_urls | No | — | Glob patterns to exclude |

When to use: Site analysis, content auditing, planning before crawl/batch.

6. AI-Powered Answers

Web-sourced answers with citations. Optionally provide JSON schema for structured output. Synchronous.

curl -sS -X POST "https://api.olostep.com/v1/answers" \
  -H "Authorization: Bearer $OLOSTEP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task": "What are the top 5 AI agent frameworks in 2026?"
  }'

With structured output:

curl -sS -X POST "https://api.olostep.com/v1/answers" \
  -H "Authorization: Bearer $OLOSTEP_API_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "task": "Find the founders and funding of Olostep",
    "json_format": {"company": "", "founders": [], "total_funding": "", "last_round": ""}
  }'

Response: result.json_content matches your schema. result.sources lists URLs used.

When to use: Research, fact-checking, competitive analysis, structured web intelligence.

7. Retrieve Content by ID

Crawl and batch results return retrieve_id per item. Get actual content with:

curl -sS "https://api.olostep.com/v1/retrieve?retrieve_id=RETRIEVE_ID&formats=markdown" \
  -H "Authorization: Bearer $OLOSTEP_API_KEY"

Common Workflows

Research a topic

Search Google → find sources
Scrape top results → get full content
Synthesize into deliverable

Ingest documentation

Map the docs site → discover URLs
Batch or Crawl relevant sections
Retrieve content by ID

Debug an error

Search the exact error message (in quotes)
Scrape GitHub issues or Stack Overflow answers
Apply the fix

Extract structured data at scale

Map to find all product/listing URLs
Batch with parser for structured JSON
Retrieve and process results

Available Parsers

Use with "parser": {"id": "PARSER_ID"} and "formats": ["json"]:

| Parser ID | Use Case | |-----------|----------| | @olostep/google-search | Google SERP (organic, knowledge graph) | | @olostep/amazon-it-product | Amazon product pages | | @olostep/extract-emails | Email addresses from pages | | @olostep/extract-socials | Social media links |

Rules

Always check $OLOSTEP_API_KEY is set before making requests.
Default to formats: ["markdown"] — most efficient for LLM context.
Content is inside result.markdown_content (not a top-level field).
Crawls and batches are async — poll status before fetching results.
Only fetch what the current task needs. Do not scrape unnecessarily.