Crawl API, Billing Hardening, and Blog Enhancements
Launches the Crawl API with full playground support, hardens billing with fraud detection and spend limits, and adds a rich blog experience with search, reactions, and author profiles.
The Crawl API is now live with POST/GET/DELETE endpoints, progressive depth crawling, and full playground support. Crawl results include per-page cost breakdowns and automatic refunds for failures.
Both Python and Node.js SDKs now include crawl methods. Start crawls, poll status, and retrieve results programmatically.
Export scrape results directly to S3, GCS, or Azure. Configure storage integrations in Settings and attach them to batch and schedule jobs.
New Features
7Crawl API with playground
The Crawl API is now live with POST/GET/DELETE endpoints, progressive depth crawling, and full playground support. Crawl results include per-page cost breakdowns and automatic refunds for failures.
Crawl SDK methods
Both Python and Node.js SDKs now include crawl methods. Start crawls, poll status, and retrieve results programmatically.
Crawl billing transparency
Crawl costs now appear in your usage breakdown and transaction history. The crawl status page shows tier breakdowns and session cost tracking.
Forgot password flow
You can now reset your password via a secure email flow. The forgot password link on the sign-in page walks you through the recovery process.
Cloud storage export
Export scrape results directly to S3, GCS, or Azure. Configure storage integrations in Settings and attach them to batch and schedule jobs.
Per-key and account spend limits
You can now set spending limits per API key and at the account level. Scheduled and batch scrapes enforce these limits, and credits are automatically refunded when jobs fail.
Interactive pricing tier cards
Tier cards on the billing page are now interactive, making it easier to understand and compare what each tier offers.
Improvements
1Faster page load times
Reduced initial page load time by deferring below-fold JavaScript and heavy 3D animations. Dashboard skeleton loaders replace spinners for smoother perceived performance.
Bug Fixes
3OAuth sign-in loop fixed
Fixed an issue where some users experienced an infinite sign-in loop when using Google OAuth due to a database table mismatch.
Dashboard balance link added
The balance area in the dashboard header is now clickable and links directly to the billing page.
Crawl max limits increased
Crawl limits have been raised to 100,000 pages and 50 levels of depth, supporting larger site crawls.
Plus 202 internal changes for stability and performance.