Give Your Agent Live Web Access with Crawlbase

Research & Learning

AI agents are only as good as the data they can reach. Training data ages, APIs cover a fraction of the web, and modern sites hide behind JavaScript rendering and anti-bot walls. Crawlbase MCP connects your OpenClaw agent to the live web through one MCP server, backed by crawling infrastructure trusted by 70,000+ developers.

Powered by Crawlbase — web data infrastructure trusted by 70,000+ developers. Every tool call rides rotating residential and datacenter proxies, JavaScript rendering, retries, and success-based billing — failed requests are never charged.

What the skill gives your agent

crawl

Fetches any URL and returns clean HTML, with JavaScript rendering and smart proxy rotation handled for you.

crawl_markdown

Returns the page as LLM-ready Markdown, ideal for feeding context windows and RAG pipelines.

crawl_screenshot

Captures a full-page screenshot, useful for visual checks and UI-aware agents.

Setup

1
Sign up at Crawlbase and grab your free Normal and JavaScript tokens (no card required). crawlbase.com

Add the MCP server to your OpenClaw configuration:

openclaw config (JSON)

{
  "mcpServers": {
    "crawlbase": {
      "type": "stdio",
      "command": "npx",
      "args": ["@crawlbase/mcp@latest"],
      "env": {
        "CRAWLBASE_TOKEN": "your_token_here",
        "CRAWLBASE_JS_TOKEN": "your_js_token_here"
      }
    }
  }
}

3
Restart your agent. The three tools appear automatically. See the crawlbase-mcp repository for the full tool reference. crawlbase/crawlbase-mcp

Use cases

Research that stays current

Ask your agent to compare products, summarize news, or monitor competitors. Instead of answering from stale training data, it calls crawl_markdown and reasons over what the page says right now.

Documentation into context

Point the agent at docs pages, changelogs, or knowledge bases. crawl_markdown strips the chrome and hands back clean Markdown that slots straight into the model's context or your vector store.

Sites that fight back

E-commerce listings, travel prices, real estate portals: pages that need JavaScript rendering and rotate defenses against plain HTTP clients. The JavaScript token routes the request through a real browser with anti-bot handling, so the agent gets the rendered page instead of a challenge wall.

Visual verification

Building an agent that checks layouts, confirms a deploy, or documents what a page looked like at a point in time? crawl_screenshot returns the evidence as an image the agent can attach or analyze.

Starter Prompts

"Fetch our competitor's pricing page as Markdown and summarize what changed since last month."

"Crawl these three docs pages and turn them into clean Markdown for my knowledge base."

"Take a full-page screenshot of our landing page and check whether the deploy looks right."

OpenClaw Use Cases