Benchmarks
Krawl's published scores on FRAMES, SimpleQA, and FinSearchComp, plus the Krawl Crypto Research Bench — methodology, leaderboard, and scoring code.
This page is the audit trail for any "Krawl beats X" claim. Every score below is dated, has the run config it was generated from, and links to the scoring code on GitHub. If a number on a marketing page disagrees with a number here, this page wins.
Disclosure. We publish the methodology and the harness, but we are not a neutral third party. Re-run the harness yourself to verify; the code is MIT and the eval data is public. See Trust for failure modes and refund policy.
Public benchmarks
FRAMES (multi-hop QA)
FRAMES is Google's multi-hop QA bench (824 questions, multi-doc reasoning). The benchmark is the right hardness ceiling for an open-web research agent.
| Configuration | Score (0–1) | Date | Harness |
|---|---|---|---|
krawl-deep (Opus 4.7 synthesis, breadth 4, depth 3) | publishing | eval pending | scripts/bench/frames.py |
krawl-quick (Sonnet 4.6 flat loop) | publishing | eval pending | same harness, --mode quick |
Scores publish on the next eval run; the harness is checked in. Run it yourself:
git clone https://github.com/ravidsrk/krawl && cd krawl
export KRAWL_API_KEY=krawl_demo_xxx
python scripts/bench/frames.py --mode deep --limit 824SimpleQA (factual QA)
SimpleQA is OpenAI's adversarial single-hop bench (4,326 questions). Used as a freshness + factuality canary — short questions, hard sources.
| Configuration | Score (0–1) | Date | Harness |
|---|---|---|---|
krawl-quick | publishing | eval pending | scripts/bench/simpleqa.py |
krawl-deep | publishing | eval pending | same harness, --mode deep |
FinSearchComp (financial research)
FinSearchComp tests financial research over heterogeneous sources (filings, news, prices). The right bench for the finance / crypto mode mix.
| Configuration | Score (0–1) | Date | Harness |
|---|---|---|---|
krawl-crypto (crypto mode) | publishing | eval pending | scripts/bench/finsearchcomp.py |
The numbers will land here once the eval run completes; we do not prepublish "expected" scores.
Krawl Crypto Research Bench
Public benchmarks are general-purpose. Crypto research has its own failure modes — claims that flip with on-chain state, sources that disagree on supply numbers, narratives that move faster than the eval loop. The Krawl Crypto Research Bench (KCRB) is our published bench for that surface.
Why a new bench
FRAMES and FinSearchComp don't cover any of:
- On-chain freshness — TVL, holder counts, active addresses that change daily.
- Multi-source disagreement — CoinGecko vs DeFi Llama vs the protocol's own dashboards.
- Narrative tracking — claims that depend on the X/farcaster discourse around an event.
KCRB tests for those three explicitly.
Methodology
- Questions: 400 dated questions, 100 per category — token fundamentals, protocol mechanics, on-chain activity, narrative tracking.
- Sources: each question carries a
verified_answerand averification_urlset captured on the question'sas_ofdate. Both are public. - Scoring: 0–1 per question, breakdown:
factual— does the answer match the verified answer? (0/1)cited— does every claim have an[n]that resolves to a fetched source? (0/1)freshness— was at least one source dated within the question's freshness window? (0/1)disagreement— for multi-source questions, did the report surface the disagreement explicitly? (0/1)
- Aggregate: weighted mean (0.5 factual, 0.2 cited, 0.2 freshness, 0.1 disagreement). The weights are fixed in the harness and printed in the result header.
Harness + scoring code
The bench is open-source MIT. Code, eval data, and the scoring rubric all live in-repo:
- Harness:
scripts/bench/kcrb.py - Eval data:
scripts/bench/kcrb_questions.jsonl - Scoring:
scripts/bench/kcrb_score.py - Leaderboard JSON:
scripts/bench/kcrb_leaderboard.json
Run it yourself:
export KRAWL_API_KEY=krawl_demo_xxx
python scripts/bench/kcrb.py --mode crypto --limit 400
python scripts/bench/kcrb_score.py --runs runs/*.jsonlLeaderboard (KCRB v0.1)
The leaderboard is reproducible from the artifacts above. Submit a run
by opening a PR with a runs/<system>-<date>.jsonl file from the
harness.
| System | Date | Factual | Cited | Freshness | Disagreement | Aggregate |
|---|---|---|---|---|---|---|
Krawl (crypto mode) | publishing | eval pending | eval pending | eval pending | eval pending | eval pending |
| Manus | open invite | — | — | — | — | — |
| Exa research API | open invite | — | — | — | — | — |
| Perplexity (deep research) | open invite | — | — | — | — | — |
Where a competitor doesn't expose a deep-research API, we run their chat surface against the same questions; that footnote sticks with the score in the leaderboard JSON.
Methodology notes
- No mocked sources. All harnesses hit the real API surface (Krawl staging or the competitor's prod), no fakes, no replayed traces.
- Reproducible seeds. Where the engine exposes a seed, the harness sets it. Where it doesn't, the harness re-runs each question N=3 times and takes the modal verified answer.
- Dated. Every published score has the
as_ofdate for the eval data and therun_atdate for the system's response. Both live in the JSON. - Cost reporting. The harness logs the total
usageevent for each run; the leaderboard JSON includes per-run cost so "Krawl beat X but spent 10×" is provable.
If a number on this page contradicts code on main, the code wins.
Open a PR.
CrewAI tool
Wrap Krawl as a CrewAI tool — let an in-Crew analyst delegate deep, cited research without paying token costs in-loop.
Trust
How Krawl handles citation, verification, failure modes, refunds, and audit — the methodology page an autonomous agent's owner should read before wiring Krawl into a production pipeline.