Crawlbase for AI agents
Crawlbase crawls web pages at scale, managing crawler queues and storing results so a team can pull page data without running its own infrastructure. Connect it once through Arc0, and your agent, or Claude, ChatGPT and Cursor, can use it through one MCP endpoint, limited to what each user approved.
What agents do in Crawlbase.
Crawl a specific URL
Crawl a URL and store its result for later retrieval instead of fetching it live each time.
Check stored crawl results
Get a previously stored page's content back instead of crawling that URL again.
Check crawling usage
Check current crawling usage and limits before submitting a large batch of URLs.
8 Crawlbase actions, graded by risk.
Every Crawlbase action is tagged read, write or destructive, so one policy covers the whole app and new actions inherit the right default.
read
6Look things up. Allowed by default.
- crawlbase.get_stored_pageGet Stored Page
- crawlbase.get_stored_pagesGet Stored Pages
- crawlbase.get_storage_countGet Storage Count
- crawlbase.get_crawling_usageGet Crawling Usage
- crawlbase.list_stored_page_idsList Stored Page IDs
- crawlbase.crawl_urlCrawl URL
write
1Create and change things. Allow, or ask the user first.
- crawlbase.generate_user_agentsGenerate User Agents
destructive
1Delete, cancel or archive. Ask first, or deny outright.
- crawlbase.delete_stored_pagesDelete Stored Pages
Crawlbase in three steps.
- 01Your users connect CrawlbaseThey add their Crawlbase api key on Arc0 Connect, under your brand. It goes straight into the vault.
- 02You set the rulesReads run, writes like “generate User Agents” can wait for the user, and “delete Stored Pages” can be denied outright.
- 03Any agent can actYour agent calls Crawlbase through the Arc0 SDK or MCP, and so can Claude, ChatGPT and Cursor. Every call lands on the audit log.
await arc0.policies.set('crawlbase', { read: 'allow', write: 'ask', // generate_user_agents destructive: 'deny', // delete_stored_pages }) # Claude Code: the same connection, one URL $ claude mcp add --transport http arc0 \ https://mcp.arc0.ai/u/u_8f2
How Crawlbase connects.
Users add their Crawlbase api key on Arc0 Connect. It is encrypted in the vault, never shown to the model, and each user can rotate or revoke it at any time.
The same Crawlbase connection serves your agent over MCP and your own backend over REST and the proxy, so a user connects once. How Arc0 handles credentials →
- AUTH
- API key
- CREDENTIALS
- Per-tenant encrypted vault
- MODEL SEES
- Results only, never credentials
- AUDIT LOG
- Every call, on every plan
Use Crawlbase from any agent.
ClaudeChatGPTCursor
Codex
VS CodeOpenAI Agents SDKClaude Agent SDK
Vercel AI SDK
Mastra
LangGraphMore web data and scraping apps.
Crawlbase and Arc0, answered.
Can I use Crawlbase with Claude, ChatGPT or Cursor?
Yes. Connect Crawlbase to Arc0 once, then add your Arc0 MCP URL to Claude, ChatGPT, Cursor, Claude Code or any other remote-MCP client. Each assistant only gets the Crawlbase actions you allow.
How do users connect Crawlbase?
Users add their Crawlbase api key on Arc0 Connect. It is encrypted in the vault, never shown to the model, and each user can rotate or revoke it at any time.
Which Crawlbase actions can my agent take?
8 in total: 6 read, 1 write and 1 destructive, such as “generate User Agents”. Your policies decide which of them each agent may call.
Can I stop my agent from deleting things in Crawlbase?
Yes. Actions like “delete Stored Pages” are graded destructive. Set destructive actions to deny, or to ask so the user approves each one, and blocked calls still show up on the audit log.
Can my own backend call Crawlbase too?
Yes. The same Crawlbase connection is available over REST and through the Arc0 proxy, so your product and your agent share one connection per user.
Plug Crawlbase into your agent.
Your users connect Crawlbase once, under your brand. Your agent gets 8 actions behind your policies, with every call on the record.
Free to build · MCP + REST · Audit log on every plan