The AI Agent Toolbox: Search, Answer, and Scraping APIs Compared
Framing: There Is No Single “Best” Tool
Tools built for AI agents split into distinct functional layers, and most serious agent stacks combine two or three of them rather than relying on one vendor. The seven “search/answer” APIs below mostly compete on how they turn a query into results or a cited answer, while the seven “scraping/extraction” tools compete on how they turn a URL (or a whole site, or a live browser session) into clean data. Treating them as substitutes for a single “best search engine” misses the point — Exa and Serper solve completely different problems even though both return search results.
A useful mental model is a five-layer stack:
- Discovery — finding relevant URLs for a query (SERP-style or neural search)
- Answer synthesis — getting a cited, LLM-written answer instead of raw links
- Single-page extraction — turning one URL into clean markdown/JSON for context
- Site-scale crawling — turning a whole domain into a corpus
- Browser automation / anti-bot infrastructure — operating a real browser or getting through hard defenses to reach a page at all
The sections below cover each of 14 tools individually, then a practical “when to use which” decision guide.
Part 1: Search & Answer APIs
Perplexity Sonar / Search API
Perplexity’s developer platform offers both Sonar models (sonar, sonar-pro, sonar-reasoning-pro, sonar-deep-research) that return web-grounded, citation-bearing answers, and a raw Search API that returns ranked results plus extracted page content for a custom pipeline (Perplexity Search quickstart). Note: Sonar chat-completions are being folded into Perplexity’s Agent API, with support ending September 27, 2026 (rate limits docs).
Strengths
- One of the only vendors offering both raw retrieval and full cited-answer synthesis under one key, so a stack can start with answers and drop to raw results later (pricing, Search quickstart).
- Cheap raw retrieval at $5 per 1,000 Search API requests, and a request can bundle up to 5 queries as one billing unit (pricing).
- Agent-friendly controls: token caps on returned page content, path-level domain filters, language/country filters (Search quickstart).
- Flat rate limits across all tiers — 50 query units/sec with a 50-unit burst (rate limits).
- First-class LangChain tool support (LangChain tool integrations).
Weaknesses
- Developers report inconsistent reliability, unpredictable timeouts, and a status page that doesn’t always reflect real outages — acknowledged by Perplexity staff in the same thread (r/perplexity_ai).
- Same thread reports steep price increases and previously tier-gated structured JSON output (r/perplexity_ai).
- Sonar’s request fee stacks on top of token costs and scales with search-context size, making per-query cost less predictable than flat per-search pricing (pricing).
- New accounts start hard rate-limited (Tier 0 = 50 RPM for sonar/sonar-pro, 5 RPM for sonar-deep-research), unlocking only via cumulative spend (rate limits).
Pricing: Search API 5/1,000 requests; `sonar` tokens 1/1 per 1M in/out; `sonar-pro` 3/15 per 1M; `sonar-deep-research` 2/8 per 1M plus 5/1,000 searches; request fees 5–14 per 1,000 depending on model and search-context size; no published free tier (pricing).
Best for
- Agents that need cited, synthesized answers without building retrieval + reranking + summarization themselves.
- High-volume raw retrieval with strict token budgets via
max_tokens_per_page. - Deep multi-hop research where
sonar-deep-researchautonomously fans out dozens of searches.
Exa
Exa (formerly Metaphor Systems) is a “custom search engine built for AIs”: it trains its own embedding and agentic-retrieval models and returns token-efficient excerpts rather than whole pages (exa.ai).
Strengths
- Semantic/neural retrieval with relevance scores;
contentsreturned in the same call, skipping a separate scrape step; typed SDKs and clean error handling (dev.to comparison). findSimilar(similarity-by-URL) has no direct equivalent among SERP-style APIs (exa-js).- Category-specific indexes: company (50M+ pages), people (1B+), scholarly publications (350M+), news, financial filings (Search API guide).
- Fastest in one developer’s hands-on latency ranking (Exa > Tavily > Linkup), praised for clean markdown output (r/Rag thread).
- Exa states it “powers all parts of” the Devin coding agent, and is a supported GPT Researcher and LangChain retriever (exa.ai, GPT Researcher retrievers).
Weaknesses
- Costs stack: base search + extra results + summaries + Agent effort charges; one Hacker News commenter said pricing rose to parity with Perplexity/Gemini grounding and stopped scaling for their volume (HN).
- Some developers rank its result quality behind Tavily and Linkup, and report neural search behaving unpredictably on very specific technical queries (r/Rag thread, dev.to comparison).
- No published numeric rate limits — only 429s and queue headers on standard tiers (Search reference).
Pricing: 20 signup credit (~2,800 searches) + 10/month free; /search 7/1,000 (≤10 results); `/answer` 5/1,000; /contents 1/1,000 pages; Deep Search 12–$15/1,000 (pricing).
Best for
- Semantic discovery where keyword SERPs fail (“find companies/papers/people like this”).
- Token-budget-sensitive coding/docs agents wanting excerpt-level context instead of full pages.
- Structured data enrichment via
output_schema+ Deep Search in one call.
Tavily
Tavily sells a search/extract/crawl/map/research API built explicitly for LLM apps, described by LangChain as “a search engine built specifically for AI agents” (LangChain Tavily integration).
Strengths
- One call returns a synthesized answer, structured results, and optional images;
include_raw_contentadds full page text so search + extract happen in one round trip (dev.to comparison). - Frequently the “just works” pick in developer threads for accuracy and ease of setup, ranked 2nd for quality and latency in one hands-on comparison (r/Rag thread).
- Deepest agent-framework penetration in this group:
langchain-tavilyships Search/Extract/Crawl/Map tools, and Tavily is the default retriever in GPT Researcher (LangChain Tavily provider, GPT Researcher retrievers). - Genuinely usable free tier: 1,000 credits/month, no card (Tavily pricing).
Weaknesses
- Result quality depends on multiple upstream search backends, so ordering can vary; one developer said it “hasn’t been meeting my expectations” and switched to Google Search (r/Rag thread).
- Credit accounting is indirect (priced per credit, not per request) and production-grade rate limits require a paid plan (dev keys capped at 100 RPM, Research capped at 20 RPM even in production) (Rate limits).
max_resultscapped at 20 per search (Search endpoint reference).
Pricing: PAYGO 0.008/credit (basic search = 1 credit ≈ 8/1,000); Project 30/mo for 4,000 credits down to Growth 500/mo for 100,000 (~$5/1,000) (credits & pricing). Free: 1,000 credits/month.
Best for
- Drop-in web search for LangChain/GPT-Researcher-style agents wanting answer + sources + raw markdown in one call.
- Real-time news/current-events monitoring (
topic=news,time_range=day). - Site-scoped RAG ingestion via Map (discover URLs) + Crawl/Extract (pull clean markdown).
You.com API
You.com sells its consumer AI-search stack as developer APIs — Web Search, Contents, Answer, Research, and Finance Research — positioned as “the Search API for the agentic era” (You.com pricing).
Strengths
- Cited, source-backed long-form research as a distinct product tier, not something assembled from raw results; Research draws from 30+ sources and claims #1 on DeepSearchQA (Research API docs, pricing).
- Generous entry: 100 free queries/day plus
100 free credit, then5/1,000 calls returning up to 100 results each (pricing). - A G2 reviewer calls the Search API architecture RAG/agent-ready, citing ~300ms p99 latency and 99.99% availability at scale (G2 reviews).
Weaknesses
- The same reviewer notes vertical-index customization has a learning curve and usage-based costs are hard to predict, with pricing “not always readily available publicly” (G2 reviews).
- Research-tier costs escalate steeply: Research
12/1,000 vs Finance Research **110/1,000** (pricing). - No published rate limits or latency SLA beyond the free daily quota (pricing); consumer-facing sentiment is notably weaker than developer sentiment (Trustpilot 2.1/5 vs G2 4.4/5) (review roundup).
Pricing: Free 100 queries/day + 100 credit; Web Search 5/1,000; Contents 1/1,000 pages; Answer 5/1,000; Research 12/1,000; Finance Research 110/1,000 (pricing).
Best for
- Citation-heavy research report generation from many sources (academic/market/competitive research agents).
- Financial research needing filings, macro data, and traceable citations with period/unit correctness.
- Latency-sensitive RAG retrieval where LLM-ready snippets beat raw SERP parsing.
Brave Search API
Brave sells programmatic access to its own independent web index — not a Google/Bing reseller — split between classic search verticals, an LLM-context endpoint, and an OpenAI-compatible grounded Answers endpoint (Brave Search API).
Strengths
- Genuine index independence — results differ meaningfully from Google/Bing, useful for diversification, bias-testing, and cross-validation (dev.to comparison).
- Cheap and predictable:
5/1,000 requests at 50 req/sec, plus5 free credits monthly (Brave pricing docs). - Offers both raw
/llm/contextretrieval and cited answers from one vendor, OpenAI SDK-compatible (grounding docs). - Enterprise option with Zero Data Retention and custom NDAs (Brave pricing docs).
Weaknesses
- Smaller index coverage than Google/Bing; niche technical queries can return sparse results — a real gap as a primary source for general research agents (dev.to comparison).
- No semantic/neural search mode — purely traditional keyword search (dev.to comparison).
- Answers is throttled to 2 requests/sec and bills tokens on top of per-query fees (Brave pricing docs).
Pricing: Search 5/1,000 requests, 50 req/sec, 5/month free credits; Answers $4/1,000 queries + token costs, 2 req/sec (Brave pricing docs).
Best for
- Privacy/compliance-constrained agents avoiding Google/Bing-derived results.
- A cheap, high-QPS secondary retriever for cross-validating another engine’s results.
- Chatbots wanting grounded, cited answers via the OpenAI SDK with no retrieval code.
Kagi Search API
Kagi exposes its premium, ad-free search product to developers: a Search API drawing on many sources including its own in-house indexes with re-ranking, an Extract API returning LLM-ready markdown, FastGPT (fast cited answers), and a Universal Summarizer (Kagi API docs).
Strengths
- Distinct, quality-curated result set, re-ranked “to surface accurate answers and interesting finds,” with per-account domain ranking, custom URL rules, and Lenses supported through the API (Kagi API docs, API pricing).
- Cheap markdown extraction ($4/1,000 pages) ready for LLM ingestion (API pricing).
- Aggressive caching — repeat FastGPT answers and repeat-URL summaries are free (FastGPT docs).
Weaknesses
- The most expensive general search API in this comparison at
12/1,000 requests, versus5 for Brave/Perplexity/You.com and roughly $1 for Serper (API pricing, serper.dev). - No free tier on any endpoint; usage is prepaid and invoiced every 30 days or at $100 (API pricing).
- No published rate limits anywhere in the docs, and Kagi’s most-praised consumer feature — its AI Assistant/Research Assistant — has no API at all (r/SearchKagi discussion).
Pricing: Search 12/1,000 requests; Extract 4/1,000 pages; FastGPT $15/1,000 queries; no free tier (API pricing).
Best for
- Quality-over-cost agents where curated, ad-free, re-ranked results matter more than price per query.
- Opinionated retrieval where lenses and domain ranking encode a “house view” of trusted sources.
- Cheap URL-to-markdown ingestion and document summarization where repeat URLs hit free caches.
SerpApi vs. Serper.dev (SERP scraping)
SerpApi runs each request through a full browser with global IPs and CAPTCHA solving, returning Google (and 80+ other engines’) results as structured JSON (serpapi.com). Serper.dev is the same category stripped down to Google only, priced far lower (serper.dev).
Strengths (SerpApi)
- Unmatched SERP feature coverage: Knowledge Graph, Local/Maps, Shopping, Patents, AI Overview, exact pixel position, 80+ engines (Search API docs).
- Enterprise comfort: Legal Shield (up to $2M coverage), ZeroTrace Mode, SOC 2/3, ISO 27001, uptime SLAs (pricing).
- Only successful searches are billed; broad SDK and MCP support, plus LangChain and GPT Researcher integrations (integrations).
Weaknesses (SerpApi) / How Serper differs
- Price is the dominant complaint: roughly 15x Serper’s entry cost per query, and about 2–3x slower in third-party Google-timing benchmarks (Serper vs SerpAPI comparison).
- Zero LLM-native features on either — no embeddings, no answer synthesis, no citations; raw SERP JSON must be chunked/reranked yourself (serpapi.com).
- Both are structurally dependent on scraping Google, a shared business risk (dev.to comparison).
- Serper is Google-only (no 80+ engine coverage) and has no semantic/neural mode, but claims 1–2 second responses and “up to 10x cheaper” than SerpApi/Bright Data (serper.dev).
Pricing: SerpApi Free 250 searches/mo → Starter 25/mo for 1,000 → Big Data 275/mo for 30,000 (pricing). Serper 1.00/1,000 credits down to 0.30/1,000 at volume, with 50–300 QPS by pack (serper.dev).
Best for
- Serper: cheap, high-volume Google SERP retrieval inside agent loops.
- SerpApi: rank tracking, local/Maps, Shopping, and patent extraction requiring exact geo-fidelity, or regulated/enterprise deployments needing legal indemnity and SOC 2/ISO certification.
Search & Answer API Comparison Table
| Tool | Type | Free Tier | Starting Paid Price | Standout Feature |
|---|---|---|---|---|
| Perplexity Sonar / Search API | AI answer + SERP-style retrieval | None stated | $5/1,000 requests | Cited answer synthesis and raw retrieval under one key |
| Exa | Neural/semantic search + AI answer | 20 credit + 10/mo | $7/1,000 /search | Embedding-based retrieval with findSimilar and category indexes |
| Tavily | AI-agent search + crawler | 1,000 credits/mo | ~$8/1,000 PAYGO | Search + answer + raw markdown in one call, plus Extract/Crawl/Map |
| You.com | SERP-style + AI answer + deep research | 100 queries/day + $100 credit | $5/1,000 | Tiered research stack up to Finance Research |
| Brave Search API | Independent index + LLM context + answer | $5/mo credits | $5/1,000 | Fully independent index, OpenAI-compatible Answers |
| Kagi | Curated search + answer + summarizer | None | $12/1,000 | Re-ranked premium results with Lenses/domain-ranking control |
| SerpApi / Serper.dev | SERP scraping | 250/mo (SerpApi) vs ~2,500 (Serper) | 25/mo for 1,000 (SerpApi) vs 1/1,000 (Serper) | SerpApi: fidelity + compliance. Serper: same data at ~1/15 the price |
Part 2: Scraping, Extraction & Browser Infrastructure
Firecrawl
Firecrawl is a hosted crawler and managed browser that handles proxies, anti-bot defenses, and JS rendering, returning markdown-first “LLM-ready output” plus structured JSON (Firecrawl docs).
Strengths
- Full lifecycle in one API: Scrape, Search (search + full content in one call), Map (URL discovery), Crawl, Parse (PDF/DOCX/XLSX → markdown/JSON), Agent, and a Browser Sandbox for agentic clicking/form-filling (Firecrawl docs).
- Widest agent-framework list of any tool here: LangChain, LlamaIndex (
FireCrawlWebReader), CrewAI, CAMEL-AI, Praison AI, plus Claude Code, Codex, Cursor, Windsurf (integrations). - Failed scrapes (no result) aren’t charged (pricing).
Weaknesses
- Self-hosting is widely panned by developers as buggy and poorly documented, with several moving to crawl4ai instead (r/LocalLLaMA).
- Cost complaints dominate community feedback, and structured-extraction formats (JSON/Question/Highlight) add 4 credits per page on top of the base cost (pricing).
- Credits don’t roll over by default (billing docs).
Pricing: Free 1,000 credits/mo, 2 concurrent; Hobby 19/mo for 5,000 credits; Standard 83/mo (annual) for 100,000; up to Scale $599/mo for 1,000,000 (pricing).
Best for
- One API for “give my agent the web”: search + scrape + crawl + map + interact under one credit balance.
- Building a RAG corpus from an entire documentation site (Map → Crawl → markdown).
- Mixed web + local document ingestion via Parse.
Jina AI Reader & Search
Jina AI (acquired by Elastic in October 2025) offers Reader (r.jina.ai), Search (s.jina.ai), and DeepSearch — markdown-first URL-to-LLM-text conversion usable by simply prefixing a URL (Jina Reader).
Strengths
- Lowest-friction reader in the category — no API key needed for basic use (Jina Reader).
- Deep controls: CSS-selector targeting, iframe/shadow-DOM extraction, cookies, proxies, locale, and preprocessing JS (API docs).
- DeepSearch is OpenAI-compatible, iterates search→read→reason, and returns URL citations (DeepSearch).
- Same URL cached for 5 minutes; one shared token balance across Reader/Search/Embeddings/Reranker (Jina Reader).
Weaknesses
- No official price table (the public pricing page 404s); third-party trackers estimate ~$0.05/1M tokens (Markaicode).
- The 10M free tokens are a one-time grant, not monthly, shared across all Search Foundation products (Markaicode).
- Open GitHub issues report 503 errors, missing images, and inconsistent output on paid keys (reader issues).
- Single-URL only — no crawl/sitemap product (API docs).
Pricing: 10M free tokens per key (one-time), keyless basic Reader; billed on tokens thereafter, ~$0.05/1M indicative (Jina Reader, Markaicode).
Best for
- One-off URL-to-markdown conversion for LLM context with zero integration.
- A full RAG retrieval stack on one key (Search → Reader → Embeddings → Reranker).
- Drop-in, cited deep research inside an OpenAI-compatible client.
Diffbot
Diffbot converts arbitrary pages into structured JSON using computer vision and NLP against a standard page-type ontology — not LLM prompting (Diffbot Extract).
Strengths
- Zero-config extraction via a single GET call, with page-type APIs for Article, Product, Image, Video, Discussion, plus beta Event/List/Job (Extract API docs).
- The Knowledge Graph is the true differentiator: 10B+ self-updating entities fused from many sources, queryable via a structured query language (Knowledge Graph).
- Crawling itself costs 0 credits (pricing).
- AWS reviewers note Diffbot’s ML-based extraction is more stable long-term than rule-based scrapers, since it isn’t broken by site redesigns (AWS Marketplace review).
Weaknesses
- Highest paid-tier floor in this comparison at $299/mo, called out of reach for small teams (pricing, Puzzleinbox review).
- Confusing credit semantics: 1 credit/page vs. 25/KG entity vs. 100/facet record (pricing).
- No browser actions, no markdown output — JSON only, and no published anti-bot/CAPTCHA handling (Diffbot Extract).
Pricing: Free 10,000 credits/mo, 5 req/min; Startup 299/mo for 250,000 credits; Plus 899/mo for 1,000,000 (pricing).
Best for
- Schema-consistent article/product extraction at scale without per-site parsers.
- Entity enrichment and market/competitor monitoring from a pre-crawled graph.
- GraphRAG pipelines (Diffbot NLP → Neo4j via LangChain).
Apify
Apify is a scraping and automation platform built on “Actors” — pre-built or custom scraper programs — with 67,742 ready-made Actors in its store (Apify About).
Strengths
- Marketplace breadth means no code for common targets (social platforms, marketplaces, YouTube) (Apify About).
- Richest agent-integration surface: an Apify MCP server, connectors for OpenAI Agents SDK, LangChain, LangGraph, Vercel AI SDK, Google ADK (AI integrations).
- Real platform primitives — datasets, key-value stores, request queues, webhooks, schedules — not just a request/response endpoint (pricing).
Weaknesses
- Compute-unit + storage + transfer metering is hard to predict, and one reviewer called it “more expensive than anticipated” (r/automation).
- Third-party Actor quality varies — configured item limits are sometimes ignored, leading to unexpected billing (r/automation).
- Heavyweight for simple jobs — spinning up a full Actor is overkill when a plain API endpoint would do (r/WebScrapingInsider).
Pricing: Free 5/mo prepaid usage (0.20/CU); Starter 19/mo + PAYG; up to Business 999/mo ($0.13/CU) (pricing).
Best for
- Scraping popular platforms where a maintained Actor already exists.
- Giving an MCP-based or LangChain agent a large library of web-data tools without building scrapers.
- Scheduled, stateful production pipelines needing queues and webhooks.
Bright Data
Bright Data wraps a very large proxy network (residential/ISP/datacenter/mobile) in productized APIs: Web Unlocker, SERP API, Web Scraper API, and Scraping Browser (Bright Data pricing hub).
Strengths
- Deepest unblocking stack: full browser rendering, CAPTCHA solving, fingerprint/OS-level emulation, near-100% claimed success rate (Web Unlocker pricing).
- Success-only billing — no charge for failed deliveries (Web Unlocker pricing).
- Scraping Browser is Puppeteer/Playwright/Selenium-compatible with unlocking built in (Scraping Browser pricing).
- Strong enterprise reviews: 4.7/5 across 342 G2 reviews, led by support and ease of use (G2 reviews).
Weaknesses
- Cost is the dominant complaint across reviews, and one long-time user reports success rates against DataDome/Cloudflare have dropped while still billing bandwidth on failed 403 responses (r/WebScrapingInsider).
- Web Unlocker explicitly doesn’t work with Puppeteer/Playwright — that requires switching products to Scraping Browser (Web Unlocker pricing).
- Committed pricing tiers start at $499/mo per product, and no markdown/LLM-context output is published (pricing).
Pricing: Free 5,000 requests/mo per product; PAYG 1.5/1,000 requests; Scale 499/mo for ~380,000 requests then $1.3/1,000 (Web Unlocker pricing).
Best for
- High-volume collection from heavily protected sites where success rate matters more than price.
- Live, geo-targeted SERP data at scale.
- Adding unblocking to existing Playwright/Puppeteer automation via Scraping Browser.
ScrapingBee
ScrapingBee is a deliberately simple headless-Chrome-plus-proxies API — one endpoint, JS rendering on by default (ScrapingBee docs).
Strengths
- Minimal surface:
api_key+url, with real pre-scrape interaction viajs_scenario(click, wait, scroll, fill) (docs). - Dedicated endpoints beyond generic HTML: Google Search, Amazon, Walmart, YouTube, plus
ai_queryfor AI-driven extraction (credit system). - Excellent developer sentiment: 4.9/5 across 139 Capterra reviews, with docs repeatedly called clear and exceptional (Capterra).
Weaknesses
- Credit multipliers punish defaults — JS rendering being on by default surprises newcomers with a 5x credit burn, and stealth mode reaches 75 credits per call (credit system).
- SERP output is HTML-centric and needs further parsing (Capterra).
- No markdown output and no site-wide crawl/sitemap endpoint (docs).
Pricing: 1,000 free credits, no card; Hobby 19/mo; up to Enterprise tiers near 1,000–$2,400/mo (pricing).
Best for
- Straightforward JS-heavy fetches with no platform to learn.
- Declarative pre-scrape interaction sequences (accept cookie banner → click → scroll).
- Retrofitting unblocking into non-Python tooling via proxy mode.
Browserbase
Browserbase rents managed cloud Chromium browser fleets to AI agents, exposing Search, Fetch, Extract, Functions, and Agent Identity behind one API key, alongside its Stagehand SDK (Browserbase docs).
Strengths
- Purpose-built for agents rather than bulk extraction: Agent Identity handles anti-bot/CAPTCHA/auth walls; Functions run agent code with <5ms latency (Browserbase docs).
- Stagehand mixes Playwright-style methods with AI primitives
act/observe/extract(schema-based extraction from natural language), with self-healing actions when selectors break (Stagehand). - Cheap/expensive tool tiering: Search/Fetch are token-efficient non-browser calls, escalating to full sessions only when needed — matching LangChain’s Deep Agents pattern (LangChain Browserbase integration).
- Observability is the most-praised feature: session replay and live view turn debugging from a 30-minute log grep into a 30-second check (Doolpa review).
Weaknesses
- Multi-metered billing (browser hours + Search + Fetch + Extract + proxy GB + tokens) makes cost forecasting hard, and idle/retrying agents burn billable time (RawSignal profile).
- One-minute and one-MB session minimums make thousands of short parallel tasks costly (Browserbase plans docs).
- Young platform with real production reliability complaints reported by reviewers (Doolpa review).
Pricing: Free 1 browser hour, 3 concurrent, 1,000 Search + 1,000 Fetch calls; Developer 20/mo; Startup 99/mo (plans).
Best for
- Agents that must actually operate a browser — logins, form fills, multi-step checkout/portal flows.
- Natural-language, self-healing automation where CSS selectors break constantly.
- Debugging and auditing production agent runs via session replay.
Scraping & Extraction Comparison Table
| Tool | Type | Free Tier | Starting Paid Price | Standout Feature |
|---|---|---|---|---|
| Firecrawl | Crawler (+search & browser interact) | 1,000 credits/mo | $19/mo | Scrape+Crawl+Map+Search+Interact on one credit balance |
| Jina Reader/Search | URL reader + search/deep research | 10M one-time tokens | ~$0.05/1M tokens | r.jina.ai/<url> prefix, zero integration |
| Diffbot | Structured extraction + knowledge graph | 10,000 credits/mo | $299/mo | 10B+ entity self-updating Knowledge Graph |
| Apify | Scraping/automation platform | $5/mo prepaid | $19/mo | 67,742 ready-made Actors + MCP/LangChain integrations |
| Bright Data | Proxy/unblocking infra + APIs | 5,000 req/mo per product | $1.5/1,000 PAYG | Near-100%-success unblocking, pay-only-for-success |
| ScrapingBee | Scraping API (JS + proxies) | 1,000 credits | $19/mo | Declarative js_scenario click/scroll/fill engine |
| Browserbase | Browser infra for agents | 1 browser hour | $20/mo | Stagehand act/observe/extract + session replay |
Decision Guide: Matching the Tool to the Job
| Project type | Recommended tool(s) | Why |
|---|---|---|
| General agent web search with citations, minimal setup | Tavily or Perplexity Search/Sonar | Both return answer + sources + raw content in one call; Tavily has the deepest LangChain/GPT Researcher penetration, Perplexity lets you drop from cited answers to raw retrieval on the same key |
| Semantic discovery (“find me companies/papers/people like X”) | Exa | Only tool here with true embedding-based findSimilar and category-specific indexes (companies, people, publications) |
| High-volume, cheap Google SERP data (rank tracking, news snippets) | Serper.dev | ~1/15 the cost of SerpApi at comparable QPS |
| Enterprise/regulated SERP scraping needing legal indemnity | SerpApi | Legal Shield, SOC 2/3, ISO 27001 — procurement-friendly |
| Independent index / avoiding Google-derived data for compliance | Brave Search API | Genuinely separate index, Zero Data Retention option |
| Long-form, multi-source research reports (market, academic, financial) | You.com Research/Finance Research or Perplexity sonar-deep-research | Purpose-built multi-step synthesis across 30+ sources with citations |
| Curated, ad-free, “trust my house view of sources” retrieval | Kagi | Lenses and domain ranking let you encode which sources count, at a price premium |
| One-off URL → clean markdown for LLM context | Jina Reader (fastest to try, keyless) or Firecrawl /scrape | Jina needs zero setup; Firecrawl adds stronger anti-bot handling if the page resists |
| Building a RAG corpus from an entire website or docs site | Firecrawl Map + Crawl | Purpose-built Map (discover URLs) → Crawl (ingest) → markdown pipeline |
| Structured, schema-consistent extraction at scale (articles, products) without writing parsers | Diffbot | ML-based ontology classification is resilient to site redesigns, unlike CSS-selector scrapers |
| Entity/company enrichment, competitor or M&A monitoring | Diffbot Knowledge Graph | Pre-crawled 10B+ entity graph, queryable directly — no crawling needed |
| Scraping a specific popular platform (YouTube, marketplaces, social) | Apify Actor store | Tens of thousands of maintained, purpose-built scrapers already exist |
| Scraping heavily protected sites (aggressive anti-bot/CAPTCHA) at scale | Bright Data | Deepest fingerprint/OS-level emulation and CAPTCHA solving of any tool here |
| Quick JS-heavy page fetch with declarative click/scroll/fill before scraping | ScrapingBee | Simplest API surface with a built-in interaction scripting language |
| Agent that must log in, fill forms, or complete multi-step checkout/portal flows | Browserbase + Stagehand | Only tool here built around persistent, controllable, observable browser sessions with self-healing selectors |
| Debugging why an agent’s browser session failed in production | Browserbase session replay | Purpose-built observability — live view and full session recordings |
Building a Practical Starter Toolbox
For a solo developer or freelancer doing broad research across many projects, a reasonable minimal toolbox looks like:
- A default search/answer API for most “look this up and summarize with sources” needs — Tavily or Perplexity Search API, since both are cheap, well-integrated with agent frameworks, and cover the discovery-plus-synthesis layer in one call.
- A semantic search tool on standby — Exa, for the specific cases where you need “similar to this” discovery rather than keyword matching (competitor research, sourcing, literature discovery).
- A URL-to-markdown reader for quick one-off page ingestion — Jina Reader, since it needs no setup and is effectively free for light use, with Firecrawl as the fallback when a page fights back with anti-bot defenses.
- A crawler for corpus-building — Firecrawl’s Map + Crawl, reserved for the (less frequent) case of ingesting a whole site rather than one page.
- Browser automation held in reserve — Browserbase/Stagehand, only pulled in when a task genuinely requires clicking, logging in, or filling forms rather than just reading content.
This keeps monthly spend low (most of these have workable free tiers) while covering discovery, synthesis, single-page extraction, site-scale ingestion, and interactive browsing — the five layers most research and agent projects actually need.