Quickstart

Run the API

From the repository root:

make api

The API listens on http://127.0.0.1:8000 by default. For browser-first rendering, start Bee Engine in another terminal:

make install
make playwright-install
make bee-engine

Scrape a page

curl http://127.0.0.1:8000/scrape \
  -H 'Content-Type: application/json' \
  -d '{
    "url": "https://example.com",
    "formats": ["markdown", "links"],
    "use_browser": "auto"
  }'

The response contains the final URL, Markdown, selected metadata, and any optional formats requested in the body.

Start a crawl

Distributed crawls require Postgres and a worker:

make db-up
export BEECRAWL_DATABASE_URL=postgres://postgres:postgres@127.0.0.1:55432/beecrawl
make migrate-up
make api

In another terminal:

make worker

Then enqueue and poll a crawl:

curl http://127.0.0.1:8000/crawl \
  -H 'Content-Type: application/json' \
  -d '{"url":"https://example.com","limit":100,"maxDepth":2}'

curl 'http://127.0.0.1:8000/crawl/CRAWL_ID?offset=0&limit=20'

Get started

Core concepts

API reference

SDKs

Run the API

Scrape a page

Start a crawl

​Run the API

​Scrape a page

​Start a crawl

Run the API

Scrape a page

Start a crawl