Asynchronous jobs - BeeCrawl

POST /crawl and POST /batch/scrape return an ID immediately. Poll the matching status endpoint with offset and limit query parameters.

Crawl

curl http://127.0.0.1:8000/crawl \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","limit":100,"maxDepth":2,"maxRetries":2}'

curl 'http://127.0.0.1:8000/crawl/CRAWL_ID?offset=0&limit=20'

The status response includes status, total, completed, failed, page data, errors, and pagination metadata. Use DELETE /crawl/{id} to request cancellation.

Batch scrape

Batch scrape processes the submitted URLs independently and never follows links:

curl http://127.0.0.1:8000/batch/scrape \
  -H 'Content-Type: application/json' \
  -d '{"urls":["https://example.com","https://example.org"]}'

Use GET /batch/scrape/{id} to poll and DELETE /batch/scrape/{id} to cancel.