Get started

WebCrawlerAPI for AI agents

WebCrawlerAPI crawls and scrapes web pages and monitors sites for changes, used by data teams building datasets or feeds from public web content. Connect it once through Arc0, and your agent, or Claude, ChatGPT and Cursor, can use it through one MCP endpoint, limited to what each user approved.

Connect WebCrawlerAPISee the 10 actions ↓
Web data and scrapingAPI keyMCP + RESTwebcrawlerapi.com
AUDIT LOG · WEBCRAWLERAPIPOLICY: acme-support
09:41:07 · claude · u_8f2read
webcrawlerapi.list_feeds
List Feeds✓ allowed · 212ms
09:41:08 · claude · u_8f2read
webcrawlerapi.get_feed
Get Feed✓ allowed · 164ms
09:41:09 · claude · u_8f2write
webcrawlerapi.create_feed
Create Feed✓ approved · approved by user
09:41:12 · claude · u_8f2destructive
webcrawlerapi.delete_feed
Delete Feed✕ blocked · policy: deny
EVERY WEBCRAWLERAPI CALL, ON THE RECORD
01 · USE CASES

What agents do in WebCrawlerAPI.

01

Scrape a single page

Pull the content of a specific page for use in a dataset or summary.

02

Create a crawl job

Start a crawl across a site to collect pages matching a pattern.

03

Confirm before deleting a feed

Check with the user before deleting a monitored feed, since its change history goes with it.

02 · ACTIONS

10 WebCrawlerAPI actions, graded by risk.

Every WebCrawlerAPI action is tagged read, write or destructive, so one policy covers the whole app and new actions inherit the right default.

read

6

Look things up. Allowed by default.

  • webcrawlerapi.get_feed
    Get Feed
  • webcrawlerapi.list_feeds
    List Feeds
  • webcrawlerapi.get_crawl_job
    Get Crawl Job
  • webcrawlerapi.list_feed_changes
    List Feed Changes
  • webcrawlerapi.get_organization_costs
    Get Organization Costs
  • webcrawlerapi.scrape_page
    Scrape Page

write

3

Create and change things. Allow, or ask the user first.

  • webcrawlerapi.create_feed
    Create Feed
  • webcrawlerapi.create_crawl
    Create Crawl
  • webcrawlerapi.set_feed_state
    Set Feed State

destructive

1

Delete, cancel or archive. Ask first, or deny outright.

  • webcrawlerapi.delete_feed
    Delete Feed
03 · HOW IT WORKS

WebCrawlerAPI in three steps.

  1. 01Your users connect WebCrawlerAPIThey add their WebCrawlerAPI api key on Arc0 Connect, under your brand. It goes straight into the vault.
  2. 02You set the rulesReads run, writes like “create Feed” can wait for the user, and “delete Feed” can be denied outright.
  3. 03Any agent can actYour agent calls WebCrawlerAPI through the Arc0 SDK or MCP, and so can Claude, ChatGPT and Cursor. Every call lands on the audit log.
POLICY.TS
await arc0.policies.set('webcrawlerapi', {
  read: 'allow',
  write: 'ask',        // create_feed
  destructive: 'deny',  // delete_feed
})

# Claude Code: the same connection, one URL
$ claude mcp add --transport http arc0 \
    https://mcp.arc0.ai/u/u_8f2
04 · AUTH AND DATA

How WebCrawlerAPI connects.

Users add their WebCrawlerAPI api key on Arc0 Connect. It is encrypted in the vault, never shown to the model, and each user can rotate or revoke it at any time.

The same WebCrawlerAPI connection serves your agent over MCP and your own backend over REST and the proxy, so a user connects once. How Arc0 handles credentials →

AUTH
API key
CREDENTIALS
Per-tenant encrypted vault
MODEL SEES
Results only, never credentials
AUDIT LOG
Every call, on every plan
05 · WORKS WITH

Use WebCrawlerAPI from any agent.

Claude
ChatGPT
Cursor
Codex
VS Code
OpenAI Agents SDK
Claude Agent SDK
Vercel AI SDK
Mastra
LangGraph
06 · RELATED

More web data and scraping apps.

Firecrawl30 actionsTavily5 actionsExa20 actionsAgentql3 actionsAgenty79 actionsAirtop10 actionsJust One API288 actionsApify183 actions
ALL WEB DATA AND SCRAPING APPS →
07 · FAQ

WebCrawlerAPI and Arc0, answered.

Q01

Can I use WebCrawlerAPI with Claude, ChatGPT or Cursor?

Yes. Connect WebCrawlerAPI to Arc0 once, then add your Arc0 MCP URL to Claude, ChatGPT, Cursor, Claude Code or any other remote-MCP client. Each assistant only gets the WebCrawlerAPI actions you allow.

Q02

How do users connect WebCrawlerAPI?

Users add their WebCrawlerAPI api key on Arc0 Connect. It is encrypted in the vault, never shown to the model, and each user can rotate or revoke it at any time.

Q03

Which WebCrawlerAPI actions can my agent take?

10 in total: 6 read, 3 write and 1 destructive, such as “create Feed”. Your policies decide which of them each agent may call.

Q04

Can I stop my agent from deleting things in WebCrawlerAPI?

Yes. Actions like “delete Feed” are graded destructive. Set destructive actions to deny, or to ask so the user approves each one, and blocked calls still show up on the audit log.

Q05

Can my own backend call WebCrawlerAPI too?

Yes. The same WebCrawlerAPI connection is available over REST and through the Arc0 proxy, so your product and your agent share one connection per user.

Get started

Plug WebCrawlerAPI into your agent.

Your users connect WebCrawlerAPI once, under your brand. Your agent gets 10 actions behind your policies, with every call on the record.

Connect WebCrawlerAPIBrowse all apps →

Free to build · MCP + REST · Audit log on every plan