/crawl
POST/crawl
Start an asynchronous crawl job that spiders a website and scrapes every discovered page. Returns a crawl ID and status URL for polling results via GET /crawl/{id}. Supports depth control, path filtering, sitemap strategies, and webhook notifications.
Request
Responses
- 200
- 400
- 401
- 403
- 404
- 408
- 429
- 500
- 503
The request contains errors or didn't properly encode content.
The request is missing, or contains bad, authorization credentials.
The request targets a destination that is not allowed.
Resource couldn't be found.
The request took has taken too long to process.
Too many requests are currently being processed.
An internal error occurred when handling the request.
Service is unavailable.