success envelope.
The compatibility contract is pinned to firecrawl-py==4.32.1. The scrape
adapter accepts markdown, html, rawHtml, links, screenshot, images,
summary, attributes, question, highlights, json, and
deterministicJson. The two JSON formats accept schema
and prompt; json uses the configured OpenAI-compatible provider and falls
back to deterministic extraction when no provider is configured. Screenshot
format objects support fullPage, quality, and a bounded
viewport; quality requests use JPEG while other screenshots use PNG.
Unsupported formats are rejected with JSON 400 responses instead of being silently
ignored. The defaults emitted by firecrawl-py 4.x are accepted when they
match BeeCrawl’s behavior. Unsupported behavior-changing values, including
mobile=true and fastMode=true, are also rejected explicitly.
skipTlsVerification is supported for direct scrape, search scraping, and
crawl jobs. Caller-supplied headers are forwarded by direct scrape and search
scraping and are included in the cache key. includeTags, excludeTags,
onlyMainContent, and onlyCleanContent are applied when deriving Markdown,
content HTML, and links from either a fresh or cached raw HTML snapshot.
attributes accepts Firecrawl selector/attribute pairs and returns every
matching value. question returns answer, and highlights returns
highlights; both use the configured OpenAI-compatible provider and fall back
to deterministic relevant-passage selection when no provider is configured.
Map returns the current link-object shape and uses Firecrawl’s current defaults
(limit=5000, includeSubdomains=true). Crawl defaults to limit=10000, does
not remove query parameters unless requested, and treats an omitted discovery
depth as effectively unbounded within the page limit. Crawl workers respect
robots.txt by default using FirecrawlAgent; robotsUserAgent selects a custom
matching group and request user agent, while ignoreRobotsTxt=true explicitly
bypasses the policy. Search defaults to ten
results and accepts timeouts from one second through five minutes.
includePaths and excludePaths are validated regular expressions applied to
the URL pathname. With regexOnFullURL=true, the same expressions are applied
to the complete URL. Exclusions take precedence over inclusions for both the
starting page and discovered pages.
The crawl queue persists sitemap=skip|include|only. By default discovery stays
under the starting URL’s path and host; crawlEntireDomain, allowSubdomains,
and allowExternalLinks expand that scope in that order.
delay spaces task starts by the requested number of seconds (up to 60), and
maxConcurrency caps active task leases per crawl or batch job. Both limits are
enforced atomically by the shared PostgreSQL queue across worker processes.
Query parameters participate in crawl identity unless
ignoreQueryParameters=true; hash-router fragments are retained while ordinary
anchors are removed. deduplicateSimilarURLs=true additionally coalesces
http/https, www, trailing-slash, and index-document aliases through a
database-enforced per-job key.
POST /v2/crawl accepts a UUID x-idempotency-key. The first request reserves
the key atomically; reuse returns HTTP 409 even after the associated crawl has
expired.
Crawler and batch requests accept Firecrawl’s string or object webhook
configuration and emit started, page, and completed lifecycle events.
Set BEECRAWL_WEBHOOK_HMAC_SECRET; every JSON request is signed as
X-Firecrawl-Signature: sha256=<HMAC-SHA256> and custom metadata and headers
are preserved. Event claims are persisted so distributed workers do not emit
duplicate lifecycle callbacks.
The crawl and batch status URLs also accept WebSocket upgrades used by the
official watcher clients. A socket receives an initial catchup status, a
document message for each newly completed page, and a terminal done
message. Large catchups are assembled across all database result pages.
Set BEECRAWL_PROXY_URL to an HTTP(S) or SOCKS5 URL (credentials may be
embedded) to enable proxy=basic; proxy=auto uses it when configured. The
same proxy is applied to static HTTP and Playwright traffic, persisted for
crawl and batch workers, forwarded by search result scraping, and included in
the scrape cache key.
BEECRAWL_STEALTH_PROXY_URL and BEECRAWL_ENHANCED_PROXY_URL enable the
matching Firecrawl proxy modes. auto selects basic, enhanced, then stealth in
that order based on configured capacity. Stealth/enhanced browser contexts also
install automation-evasion navigator shims and report residential proxy use.
For non-rendered requests, BEECRAWL_TLS_CLIENT_URL enables an HTTP fetch
service that impersonates the browser profile selected by
BEECRAWL_TLS_CLIENT_PROFILE (default chrome_124). The adapter forwards
headers, proxy credentials, timeout, and TLS policy, and safely falls back to
the built-in HTTP client when the service is unavailable.
Scrape metadata reports the selected engine, ordered engineOutcomes, an
optional fallbackReason, and proxyUsed. These fields expose automatic
browser/TLS/static fallback decisions on scrape, batch, crawl, and webhook
documents.
In auto browser mode, Beecrawl runs the browser and HTTP extraction paths in
parallel, scores substantive text, document structure, HTTP status, and common
bot-challenge markers, then returns the higher-quality candidate. Screenshot
requests remain browser-only because the HTTP engine cannot produce images.
For distributed rendering, set BEE_ENGINE_URLS to a comma-separated pool.
The API ranks instances by the free page capacity reported by /health and
rolls failed renders over to the next instance. Health responses also expose
the instance ID, active and available pages, job counts, engine availability,
and service version.
POST /v2/scrape accepts ordered browser actions: duration or selector
wait, click, write, press, scroll, screenshot, scrape,
executeJavascript, and pdf. Action responses contain Firecrawl-compatible
grouped screenshots, scrapes, and javascriptReturns, plus pdfs and an
ordered results array. Actions always use Bee Engine and bypass the scrape
cache. Requests are limited to 50 actions, a 300-second total timeout, 256 KiB
of script/text payload, 4 KiB selectors, and cumulative waits within the
request timeout.
To smoke-test scrape, map, parse, search, crawl, batch, and error handling
through the pinned official Python SDK while a local API is running:
POST /v2/parse accepts multipart HTML/XHTML, PDF, DOC/DOCX, ODT, RTF, and
XLS/XLSX uploads. Send [email protected] and, optionally, an options JSON
part. PDF parsers accept mode: "fast" | "auto" | "ocr" and maxPages from
1 to 10000; auto applies Tesseract OCR to pages without usable embedded
text. Supported output formats are markdown, html, rawHtml, summary,
and json. The Bee Engine documents extra supplies Office parsers and its
container includes Antiword and Tesseract.
For clients that cannot send multipart bodies, POST /v2/parse/base64 accepts
JSON instead:
data is accepted as an alias for base64; bare Base64 and document data
URLs are valid. Every decoded document is limited to 50 MB.
For two-step uploads, call POST /v2/parse/upload-url with filename, upload
the bytes to the returned expiring PUT URL, then send uploadRef and parse
options to POST /v2/parse/reference. Upload references expire after 15
minutes, are capped at 50 MB, and are consumed once. Multipart clients may
also submit the reference as an uploadRef field to /v2/parse.
Search supports web, news, and images sources. Web results can be
restricted with GitHub, research, and PDF categories plus includeDomains or
excludeDomains; lang, country, location, and tbs are forwarded to
providers. Bing RSS and image search back the news and image sources, while
web search uses configured SearXNG with DuckDuckGo fallback. Set
asyncScraping: true to scrape up to eight results concurrently and
highlights: true to return query-relevant passages from scraped content (or
provider snippets when content is unavailable).
Persistent browser sessions are available under both /v2/browser and the
/v2/interact alias. A session keeps one Playwright context and page alive, so
cookies, local storage, and page JavaScript state survive executions. Node
executions run in the page context; replay endpoints list up to 50 timestamped
page snapshots and return their screenshots. Sessions default to a 10-minute
TTL and 5-minute inactivity TTL, are isolated by API key, limited by
BEE_ENGINE_MAX_SESSIONS (default 8), and cleaned every 30 seconds.
Browser-rendered scrape results retain a four-hour handoff snapshot. Use the
returned metadata.scrapeId with POST /v2/scrape/{scrapeId}/interact to
create a session initialized with the final URL, cookies, and browser storage.
Agent workflows are Postgres-backed asynchronous jobs. Create one with
POST /v2/agent using prompt, optional seed urls, and maxCredits; poll or
cancel it at /v2/agent/{id}. Without seeds the prompt drives web search. Each fetched source consumes one credit, source
documents and errors remain visible while the job runs, and ownership is
isolated by API key.
Monitors use POST /v2/monitor with name, url, and scheduleSeconds
(minimum 60). The worker schedules enabled monitors, while
POST /v2/monitor/{id}/run queues an immediate check. The checks endpoint
retains the complete snapshot, a line-oriented git-style text diff, and a
path-based JSON change list for the latest 100 runs. Optional monitor webhooks
use the same HMAC signature as crawl delivery and emit monitor.completed or
monitor.failed events. Agent and Monitor workflows require Postgres,
migrations, and the standard BeeCrawl worker.
Batch scrape uses the same Postgres-backed queue and worker as crawl. Crawl and
batch status responses include id, expiresAt, and a next URL while more
results are available. Error endpoints return terminal page errors separately.
The compatibility layer does not implement hosted Firecrawl usage accounts.
Request fields not implemented by an adapter are rejected rather than accepted
as no-ops.