# Krawl — full docs text

> One-shot flat dump of every published docs page for LLM ingestion.
> Authoritative copy lives at https://docs.krawl.sh; this file is generated
> at build time from `docs/content/docs/*.mdx`.

Generated: build-time. Schema: see https://docs.krawl.sh/openapi.json.


# Krawl

Source: https://docs.krawl.sh/docs/index

Krawl is an autonomous deep-research API that performs **recursive breadth×depth tree search**, synthesizes findings into a cited report with source verification, and streams progress via SSE events. It is built for AI agents: drop in a skill, authenticate with an API key, or let an autonomous agent pay per call with **x402** — no account, no signup.

Built with FastAPI and litellm for multi-provider LLM routing. No agent framework — raw tool-calling with a 4-tier model strategy across AWS Bedrock and Anthropic, with cross-provider fallback to survive an outage.

## Key Capabilities

- **Recursive research tree** — parallel queries at each depth level, learning extraction, gap detection, automatic recursion
- **STORM perspective discovery** — multi-perspective research inspired by Stanford's STORM methodology
- **Multi-pass synthesis** — gap check → revise cycles with quality scoring (Opus 4.7 for final synthesis)
- **4-tier model strategy** — Haiku for high-volume tasks, Sonnet for research/planning, Opus for synthesis, with cross-provider fallback
- **Cross-session memory** — knowledge base that persists learnings across research sessions
- **9 research modes** — deep, quick, crypto, token-analysis, protocol-research, whale-tracking, narrative, risk-assessment, yield-strategy
- **Document upload** — PDF, TXT, MD, CSV, JSON — up to 5 files, 10MB each
- **8 research templates** — pre-built query patterns for common research tasks
- **Research steering** — inject instructions mid-stream to adjust research direction
- **Async batch tasks** — submit a research job and poll for the result, no SSE connection held open
- **Structured output** — JSON Schema-driven extraction alongside the markdown report
- **Embedded charts** — crypto time-series charts rendered from real data
- **Export** — PDF, HTML, Markdown, DOCX, and reveal.js slides
- **Lookouts** — scheduled recurring research with change detection and webhooks
- **Audit trail** — full log of every search, extraction, and synthesis step
- **27 SSE event types** — real-time streaming of every research phase

## Agent-Native by Design

- **API keys with plans** — multi-user keys, demo keys, per-request cost ceilings, monthly quotas, and usage metering. See [Authentication](/docs/authentication).
- **Keyless x402 payments** — an autonomous agent pays per request in USDC on Base. No account, no key handoff. $1.00 per quick krawl, $5.00 per deep krawl. See [Payments & x402](/docs/payments-x402).
- **MCP server** — connect Krawl as a Model Context Protocol tool, with unauthenticated discovery and gated tool calls.
- **OpenAI-compatible shim** — point any OpenAI client at `/v1/chat/completions` and get a cited research report back.
- **Discovery** — `agent.json`, `skill.md`, MCP manifest, and `/.well-known` resources for autonomous agents. See [Agents & MCP](/docs/agents-mcp).

## Architecture at a Glance

```
Client POST /research { query, mode, breadth, depth_levels }
  │
  ├─ Phase 1: Planning (Sonnet 4.6)
  │    └─ Structured research plan with topics and tasks
  │
  ├─ Phase 1.5: STORM Perspective Discovery (Haiku 4.5)
  │    └─ 3-5 research perspectives with key questions
  │
  ├─ Phase 2: Recursive Tree Search
  │    ├─ Generate N queries (Haiku 4.5) × depth levels
  │    ├─ Execute searches in parallel (Exa, X, GitHub, crypto, etc.)
  │    ├─ Extract learnings (Sonnet 4.6)
  │    ├─ Detect knowledge gaps (Sonnet 4.6)
  │    └─ Recurse with depth-1, breadth/2 if gaps remain
  │
  ├─ Phase 3: Verified Synthesis (Opus 4.7, 1 call)
  │    ├─ Gap check → optional re-synthesis
  │    └─ Multi-pass review → revise loop (up to 2 passes)
  │
  └─ SSE stream: plan → perspectives → queries → sources → learnings → report_chunk → result
```

## Quick Start

```bash
curl -X POST https://api.krawl.sh/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"query": "What is the current state of AI agents?", "mode": "deep"}'
```

The response is an SSE stream. See [SSE Events](/docs/sse-events) for all event types.

No key? Grab a demo key with no signup:

```bash
curl -X POST https://api.krawl.sh/auth/demo-key
```

## Pages

- [Getting Started](/docs/getting-started) — Setup, env vars, first request
- [Authentication](/docs/authentication) — API keys, plans, demo keys, quotas, and usage
- [Payments & x402](/docs/payments-x402) — Keyless per-request agent payments in USDC on Base
- [Agents & MCP](/docs/agents-mcp) — MCP server, OpenAI-compatible shim, and agent discovery
- [Architecture](/docs/architecture) — Deep dive into the research pipeline
- [Models](/docs/models) — 4-tier model strategy, provider routing, circuit breaker
- [Research Modes](/docs/research-modes) — All 9 modes explained
- [API Reference](/docs/api-reference) — Every endpoint, grouped by area
- [SSE Events](/docs/sse-events) — All 27 event types with schemas
- [Crypto Tools](/docs/crypto-tools) — Crypto-specific tools and modes
- [Templates](/docs/templates) — 8 pre-built research templates
- [Memory](/docs/memory) — Cross-session knowledge base
- [Export](/docs/export) — PDF, HTML, Markdown, DOCX, and slides export
- [Lookouts](/docs/lookouts) — Scheduled recurring research
- [Deployment](/docs/deployment) — AWS staging and production release
- [Testing](/docs/testing) — Test suite guide
```

---


# Getting Started

Source: https://docs.krawl.sh/docs/getting-started

## Prerequisites

- Python 3.12+
- PostgreSQL (for result persistence, memory, lookouts)
- API keys for search providers

## Local Development

```bash
# Clone and install
git clone https://github.com/ravidsrk/krawl.git
cd krawl/apps/api   # the API lives in apps/api
pip install -e ".[dev]"
```

## Environment Variables

Create a `.env` file in `apps/api/` (start from `apps/api/.env.example`). Krawl uses `pydantic-settings` and loads from `.env` automatically.

### Required

| Variable | Description |
|---|---|
| `EXA_API_KEY` | Exa search API key — primary web search provider |
| `GITHUB_TOKEN` | GitHub personal access token — repo/code search |
| `COINGECKO_API_KEY` | CoinGecko Pro API key — token data, market cap, charts |

### Required for LLM (at least one)

| Variable | Description |
|---|---|
| `AWS_BEDROCK_ACCESS_KEY_ID` | AWS access key for Bedrock (primary LLM provider) |
| `AWS_BEDROCK_SECRET_ACCESS_KEY` | AWS secret key for Bedrock |
| `AWS_BEDROCK_REGION` | AWS region (default: `us-east-1`) |
| `ANTHROPIC_API_KEY` | Anthropic direct API key (fallback provider, or primary if no Bedrock) |

### Optional

| Variable | Default | Description |
|---|---|---|
| `KRAWL_DEV_KEY` | `""` | Legacy single shared key. If empty (and no multi-user keys exist), research endpoints are open |
| `XAI_API_KEY` | `""` | xAI API key for X/Twitter search via Grok |
| `XAI_BASE_URL` | `https://api.x.ai/v1` | xAI API base URL |
| `X_BEARER_TOKEN` | `""` | X API v2 bearer token (direct, pay-per-use) |
| `FIRECRAWL_API_KEY` | `""` | Firecrawl for JS-heavy site scraping |
| `TAVILY_API_KEY` | `""` | Tavily search |
| `SERPER_API_KEY` | `""` | Serper (Google search) |
| `BRAVE_API_KEY` | `""` | Brave search |
| `NANSEN_API_KEY` | `""` | Nansen on-chain analytics |
| `DUNE_API_KEY` | `""` | Dune Analytics queries |
| `LUNARCRUSH_API_KEY` | `""` | LunarCrush social metrics |
| `COINGLASS_API_KEY` | `""` | Coinglass derivatives data |
| `MESSARI_API_KEY` | `""` | Messari research data |
| `DATABASE_URL` | `""` | PostgreSQL connection string |
| `OPENAI_API_KEY` | `""` | OpenAI API key — cross-provider fallback when Anthropic/Bedrock is down |
| `OPENROUTER_API_KEY` | `""` | OpenRouter key — optional free-model override for Tier-1 budget calls |

### Payments & Billing (optional)

| Variable | Default | Description |
|---|---|---|
| `X402_PAY_TO` | `""` | Your receiving wallet address (`0x…`). Setting this enables x402 keyless payments |
| `X402_NETWORK` | `base` | `base` (mainnet) or `base-sepolia` (testnet) |
| `X402_PRICE_SHALLOW` | `1000000` | Price per quick krawl, in atomic USDC units ($1.00) |
| `X402_PRICE_DEEP` | `5000000` | Price per deep krawl, in atomic USDC units ($5.00) |
| `STRIPE_SECRET_KEY` | `""` | Stripe key for Pro subscriptions and pay-as-you-go metered billing |
| `STRIPE_PRICE_PRO` | `""` | Stripe price ID for the recurring Pro plan |
| `STRIPE_PRICE_PAYG_METERED` | `""` | Stripe price ID for metered per-krawl usage |

See [Authentication](/docs/authentication) and [Payments & x402](/docs/payments-x402) for the full picture.

### Tuning Parameters

| Variable | Default | Description |
|---|---|---|
| `MAX_STEPS` | `40` | Maximum agent tool-calling steps |
| `DEFAULT_BREADTH` | `4` | Parallel queries per depth level |
| `DEFAULT_DEPTH` | `3` | Recursion depth levels |
| `MAX_BREADTH` | `10` | Maximum breadth per level |
| `MAX_DEPTH` | `5` | Maximum recursion depth |
| `RATE_LIMIT` | `5/minute` | API rate limit (slowapi format) |
| `SYNTHESIS_TIMEOUT` | `120.0` | Synthesis LLM call timeout (seconds) |
| `SYNTHESIS_MAX_SOURCES` | `25` | Max sources included in synthesis |
| `VERIFY_CITATIONS` | `true` | Enable retrieve-then-cite verification |
| `AUDIT_TRAIL` | `true` | Enable audit trail logging |
| `LOOKOUT_MAX_PER_USER` | `10` | Max active lookouts per API key |
| `LOOKOUT_MIN_INTERVAL_MINUTES` | `60` | Minimum time between lookout runs |

### Model Overrides

You can override the default models via env vars:

| Variable | Default |
|---|---|
| `MODEL_PLANNING` | `bedrock/us.anthropic.claude-sonnet-4-6` |
| `MODEL_RESEARCH` | `bedrock/us.anthropic.claude-sonnet-4-6` |
| `MODEL_SYNTHESIS` | `bedrock/us.anthropic.claude-opus-4-7` |
| `MODEL_QUERY_GEN` | `bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0` |
| `MODEL_X_SEARCH` | `grok-4.20-0309-non-reasoning` |
| `LEARNING_EXTRACTION_MODEL` | `bedrock/us.anthropic.claude-sonnet-4-6` |
| `GAP_ANALYSIS_MODEL` | `bedrock/us.anthropic.claude-sonnet-4-6` |

## Run the Server

From `apps/api/`:

```bash
uvicorn app.main:app --host 0.0.0.0 --port 8080 --reload
```

## Health Check

```bash
curl http://localhost:8080/health
```

```json
{"status": "ok", "version": "0.2.0", "db": "ok", "uptime_seconds": 1234.5}
```

`/health` returns `200` when healthy and `503` (with `"status": "degraded"`,
`"db": "error"`) when the database is unreachable.

## First Research Request

```bash
curl -N -X POST http://localhost:8080/research \
  -H "Content-Type: application/json" \
  -d '{
    "query": "What are the latest developments in AI agents?",
    "mode": "deep",
    "breadth": 4,
    "depth_levels": 3
  }'
```

The `-N` flag disables output buffering so you see SSE events as they arrive.

## First Search Request

For a simple single-source search without the full research pipeline:

```bash
curl -X POST http://localhost:8080/search \
  -H "Content-Type: application/json" \
  -d '{
    "query": "AI agents 2025",
    "source": "exa"
  }'
```

## Authentication

Krawl accepts three forms of authentication on its research endpoints:

1. **API key** — pass `X-API-Key: krawl_live_…`. Keys carry a plan with quotas and rate limits.
2. **Demo key** — grab one with no signup via `POST /auth/demo-key`, for trying the API.
3. **Keyless x402** — an autonomous agent pays per request in USDC on Base, with no account at all (enabled when `X402_PAY_TO` is set).

```bash
# Get a no-signup demo key
curl -X POST https://api.krawl.sh/auth/demo-key

# Use it
curl -H "X-API-Key: krawl_demo_…" -X POST https://api.krawl.sh/research \
  -H "Content-Type: application/json" \
  -d '{"query": "What is EIP-4844?", "mode": "quick"}'
```

The legacy `KRAWL_DEV_KEY` is a single shared key honored on the Krawl server for backward compatibility with internal tooling. When it is unset and no multi-user keys exist, the research endpoints are open and the server logs a warning on startup.

See [Authentication](/docs/authentication) for plans, quotas, and key management, and [Payments & x402](/docs/payments-x402) for keyless agent payments.

## CORS

Default allowed origins:
- `https://krawl.sh`
- `http://localhost:5173`
- `http://localhost:3000`

Configure via `CORS_ORIGINS` env var (JSON array format).

---


# Pricing

Source: https://docs.krawl.sh/docs/pricing

Krawl has transparent, self-serve pricing. There are two ways to pay: **API key
plans** with monthly quotas, and **x402 pay-per-request** for autonomous agents
that don't want an account. No sales calls, no "contact us" gating.

## Monthly plans

| Plan | Price | Deep reports / mo | Search credits / mo | Rate limit |
|------|-------|-------------------|---------------------|------------|
| **Demo** | Free | 3 | 100 | 5 / hour |
| **Free** | **$0 / month** | 25 | 1,000 | 10 / minute |
| **Pro** | **$49 / month** | Unlimited | Unlimited | 120 / minute |
| **Pay-as-you-go** | Metered (per request) | Unlimited | Unlimited | 60 / minute |

- **Demo** — instant key via `POST /auth/demo-key`, no signup, expires ~24h.
- **Free** — the default plan for any signed-up key.
- **Pro** — flat **$49/month** subscription via Stripe. Best for steady usage.
- **Pay-as-you-go** — metered via Stripe: **$5.00 per deep report** and
  **$1.00 per quick search**, unlimited volume. A PAYG key with no billing set up
  returns `402 {"error":"billing_setup_required"}` until checkout is completed.

Upgrade with `POST /billing/checkout` — pass `{"plan":"pro"}` (default) or
`{"plan":"payg"}` — and manage your subscription with `POST /billing/portal`.

## Per-request limits

Each request has a hard LLM-cost ceiling so spend is predictable:

| Mode | Cost ceiling |
|------|--------------|
| Deep research | **$2.50** per report |
| Quick search | **$0.40** per request |

If a request would cross its ceiling, Krawl stops gathering and synthesizes a
report from what it already has (`ceiling_hit: true` in the terminal `usage` SSE
event).

## x402 — pay per request (no account)

Autonomous agents can pay per request in **USDC on Base** with no signup using
the [x402](/docs/payments-x402) protocol:

| Mode | Price |
|------|-------|
| Deep research | **$5.00 USDC** |
| Quick research | **$1.00 USDC** |

Call a research endpoint with no `X-API-Key` → receive `HTTP 402` with payment
requirements → sign an EIP-3009 USDC authorization → retry with an `X-PAYMENT`
header.

## Quotas & overage

When a metered quota is exhausted on Demo or Free, requests return `HTTP 402`
with `{"error": "quota_exceeded", "plan": "..."}` and an upgrade hint. Pro and
Pay-as-you-go are unlimited and never hit quota.

See the machine-readable plan data in [`/agent.json`](https://krawl.sh/agent.json)
and the human pricing summary at [krawl.sh/pricing.md](https://krawl.sh/pricing.md).

---


# Authentication

Source: https://docs.krawl.sh/docs/authentication

Krawl supports three ways to authenticate a request, plus a legacy single shared key honored by the server for backward compatibility. Most of the work happens through plan-scoped API keys that carry quotas, rate limits, and a per-request cost ceiling. Autonomous agents that do not want an account can pay per request with keyless x402.

## Website sign-in

People can sign in on `https://krawl.sh/auth` with email and password, and with Google, GitHub, or X when those providers are configured. **Continue with ChatGPT** appears only when `OPENAI_OAUTH_CLIENT_ID` is set. The callback to register with OpenAI is `https://api.krawl.sh/auth/oauth/openai/callback`. That sign-in creates a Krawl session. It does not by itself grant ChatGPT plan usage, and it is separate from the API key an agent sends.

## Auth Modes

Krawl accepts three forms of authentication on its research endpoints:

1. **API key**: a plan-scoped `krawl_*` key passed in the `X-API-Key` header. Keys carry a plan with monthly quotas, a rate limit, and a per-request cost ceiling. This is the standard way to use the API.
2. **Demo key**: a rate-limited key you can mint with no signup via `POST /auth/demo-key`. It expires in 24 hours and is meant for evaluation, not production.
3. **Keyless x402**: an autonomous agent pays per request in USDC on Base, with no account at all. Enabled when the server has `X402_PAY_TO` set. See [Payments & x402](/docs/payments-x402).

A legacy single shared key (`KRAWL_DEV_KEY`) is also honored on the server for backward compatibility (see [below](#legacy-krawl_dev_key)).

## Sending a Key

Pass your key in the `X-API-Key` header on every request:

```bash
curl -X POST https://api.krawl.sh/research \
  -H "X-API-Key: krawl_live_xxxxxxxxxxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{"query": "What is EIP-4844?", "mode": "quick"}'
```

If a key is invalid, revoked, or expired, the server responds with `401`:

```json
{
  "detail": "Invalid, revoked, or expired API key"
}
```

### Key Format

Keys look like `krawl_<segment>_<token>`, for example `krawl_live_a1b2c3d4...`. The middle segment encodes the key class:

| Segment | Used by |
|---|---|
| `live` | `free`, `payg`, `pro`, `pro_zdr` plans |
| `demo` | no-signup demo keys |
| `ent` | `enterprise` plan |
| `test` | internal developer testing |

The raw key is shown **exactly once** when it is created. Krawl never stores the plaintext key, only a short display prefix (e.g. `krawl_live_a1b2c3d4e5f6`) and an HMAC-SHA256 hash of the full key. The hash is the irreversible handle used for data isolation, usage rollups, and billing linkage. Store the raw key securely; it cannot be recovered.

## Demo Keys (No Signup)

To try the API without signing up, mint a demo key. No auth is required, and the endpoint itself is rate-limited to `3/hour` per IP.

```bash
curl -X POST https://api.krawl.sh/auth/demo-key
```

```json
{
  "api_key": "krawl_demo_xxxxxxxxxxxxxxxxxxxxxxxx",
  "type": "demo",
  "plan": "demo",
  "rate_limit": "5/hour",
  "monthly_search_credits": 100,
  "monthly_deep_krawls": 3,
  "expires_in": "24 hours",
  "note": "Demo key for testing. Sign up at krawl.sh for full access."
}
```

Use it like any other key:

```bash
curl -X POST https://api.krawl.sh/research \
  -H "X-API-Key: krawl_demo_xxxxxxxxxxxxxxxxxxxxxxxx" \
  -H "Content-Type: application/json" \
  -d '{"query": "What is EIP-4844?", "mode": "quick"}'
```

The demo key is persisted (hashed) and expires 24 hours after issuance. It runs on the `demo` plan: 100 search credits, 3 deep krawls, a `5/hour` rate limit, and a `$0.50` per-request cost ceiling.

## Managing Keys

Standard key issuance and listing are **admin-only** operations, gated by a dedicated `KRAWL_ADMIN_KEY` (separate from the data-plane key). Revocation is self-service.

### Create a Key

`POST /auth/keys` issues a new key and returns the raw value once. The request body is:

| Field | Type | Default | Notes |
|---|---|---|---|
| `plan` | string | `"free"` | One of `free`, `payg`, `pro`, `pro_zdr`, `enterprise`, `test`. The `demo` plan is not issuable here (use `/auth/demo-key`). |
| `name` | string | `""` | Human-readable label, up to 100 chars. |
| `owner_email` | string | `null` | Optional owner email. |

```bash
curl -X POST https://api.krawl.sh/auth/keys \
  -H "X-API-Key: $KRAWL_ADMIN_KEY" \
  -H "Content-Type: application/json" \
  -d '{
    "plan": "pro",
    "name": "production key",
    "owner_email": "team@example.com"
  }'
```

```json
{
  "api_key": "krawl_live_xxxxxxxxxxxxxxxxxxxxxxxx",
  "prefix": "krawl_live_xxxxxxxxxxx",
  "plan": "pro",
  "rate_limit": "120/minute",
  "monthly_search_credits": -1,
  "monthly_deep_krawls": -1,
  "note": "Store this key securely — it will not be shown again."
}
```

A `-1` quota means unlimited (metered and billed). An unknown or non-issuable plan returns `400`.

### List Keys

`GET /auth/keys` returns key metadata only. It never returns raw keys. Filter by `owner_email` with a query parameter.

```bash
curl https://api.krawl.sh/auth/keys?owner_email=team@example.com \
  -H "X-API-Key: $KRAWL_ADMIN_KEY"
```

```json
{
  "keys": [
    {
      "prefix": "krawl_live_xxxxxxxxxxx",
      "plan": "pro",
      "name": "production key",
      "owner_email": "team@example.com",
      "active": true,
      "created_at": "2025-01-10T12:00:00+00:00",
      "last_used_at": "2025-01-12T09:30:00+00:00",
      "expires_at": null,
      "revoked_at": null
    }
  ]
}
```

### Revoke a Key

`DELETE /auth/keys` is self-service: it revokes the `krawl_` key presented in the `X-API-Key` header. No admin key is needed; you simply prove you hold the key you want to revoke.

```bash
curl -X DELETE https://api.krawl.sh/auth/keys \
  -H "X-API-Key: krawl_live_xxxxxxxxxxxxxxxxxxxxxxxx"
```

```json
{
  "revoked": true
}
```

Revoking a key that does not exist or is already revoked returns `404`.

## Plans & Quotas

Each key is bound to a plan that defines its monthly quotas, rate limit, per-request cost ceiling, and whether Zero Data Retention is enforced. A `-1` value means unlimited (metered and billed per use).

| Plan | Search credits / mo | Deep krawls / mo | Rate limit | Max $ / req | ZDR |
|---|---|---|---|---|---|
| `demo` | 100 | 3 | 5/hour | $0.50 | No |
| `free` (default) | 1,000 | 25 | 10/minute | $1.50 | No |
| `payg` | Unlimited (metered) | Unlimited (metered) | 60/minute | $5.00 | No |
| `pro` | Unlimited (metered) | Unlimited (metered) | 120/minute | $8.00 | No |
| `pro_zdr` | Unlimited (metered) | Unlimited (metered) | 120/minute | $8.00 | Yes |
| `enterprise` | Unlimited (metered) | Unlimited (metered) | 600/minute | $20.00 | Yes |
| `test` | 200 | 10 | 20/minute | $1.00 | No |

`free` is the default plan for any key without an explicit plan. `test` is for internal developer testing. `pro_zdr` is identical to `pro` but adds Zero Data Retention.

## How Quotas Are Enforced

Krawl enforces limits at three layers:

- **Rate limit (per-key).** The plan's `rate_limit` (slowapi format, e.g. `120/minute`) bounds request frequency. Exceeding it returns `429`.
- **Monthly quota (per-key, soft cap).** Before each research request, Krawl checks the key's monthly `deep_krawls` or `search_credits` usage against the plan limit. Finite plans that are exhausted return `402`. Unlimited plans (`-1`) skip this check. This is an eventually-consistent soft cap (no per-request lock), so a burst at the boundary can slightly overshoot.
- **Per-request cost ceiling (hard cap).** The plan's `max_cost_usd_per_request` caps LLM spend on a single request. When metered cost crosses the ceiling, the agent loop stops gathering and proceeds to synthesis with what it has, so a runaway request cannot burn unbounded money.

The metering layer (`UsageMeter`) tracks, per request, the LLM/prompt/completion **tokens**, **cost in USD**, LLM call count, search call count, and source count. On completion each request writes a usage event and bumps the monthly rollup. Billable units: a deep krawl counts as one `deep_krawl`, and `search_credits` are billed per source collected.

### Rate Limit Exceeded (429)

```json
{
  "error_code": "RATE_LIMITED",
  "message": "Rate limit exceeded",
  "detail": "Rate limit exceeded",
  "retry_after": 60
}
```

### Quota Exceeded (402)

```json
{
  "detail": {
    "error": "quota_exceeded",
    "message": "Monthly deep krawls limit reached for the free plan (25/25). Upgrade at https://krawl.sh/pricing or use pay-as-you-go.",
    "plan": "free",
    "limit": 25,
    "used": 25,
    "upgrade_url": "https://krawl.sh/pricing"
  }
}
```

## Checking Usage

### Monthly Rollup

`GET /usage` returns the caller's usage for the current month (or a given `period` like `2025-01`) alongside the plan limits and remaining quota. Requires a `krawl_` key in `X-API-Key`.

```bash
curl https://api.krawl.sh/usage \
  -H "X-API-Key: krawl_live_xxxxxxxxxxxxxxxxxxxxxxxx"
```

```json
{
  "plan": "free",
  "period": "2025-01",
  "usage": {
    "deep_krawls": 7,
    "search_credits": 142,
    "total_tokens": 318450,
    "cost_usd": 1.284512
  },
  "limits": {
    "monthly_deep_krawls": 25,
    "monthly_search_credits": 1000,
    "rate_limit": "10/minute"
  },
  "remaining": {
    "deep_krawls": 18,
    "search_credits": 858
  }
}
```

For unlimited plans, the `remaining` fields are `null` (there is no finite limit to count down from).

### Per-Request Events

`GET /usage/events` returns recent per-request usage events for the key, most recent first. Use the `limit` query parameter (default `50`, max `200`).

```bash
curl "https://api.krawl.sh/usage/events?limit=20" \
  -H "X-API-Key: krawl_live_xxxxxxxxxxxxxxxxxxxxxxxx"
```

```json
{
  "events": [
    {
      "id": "f3a1c2d4-...",
      "result_id": "9b8a7c6d-...",
      "session_id": "1a2b3c4d-...",
      "mode": "deep",
      "prompt_tokens": 41200,
      "completion_tokens": 8800,
      "total_tokens": 50000,
      "llm_calls": 23,
      "search_calls": 14,
      "source_count": 19,
      "cost_usd": 0.842301,
      "created_at": "2025-01-12T09:30:00+00:00"
    }
  ]
}
```

## Zero Data Retention

Zero Data Retention (ZDR) is available on the `pro_zdr` and `enterprise` plans. When a key's plan has ZDR enabled, the request's **query, report, sources, sessions, audit trail, and memory are NOT persisted**. Only anonymous usage counters (tokens and cost) are kept, for billing. This is intended for enterprise compliance use cases where prompt and result data must not be stored at rest.

ZDR is a property of the plan, so it applies automatically to every request made with a ZDR key. No per-request flag is required.

## Legacy KRAWL_DEV_KEY

Krawl still honors a single shared key set on the server via the `KRAWL_DEV_KEY` environment variable for backward compatibility with internal tooling. (Upgrading from a pre-2026-06 setup? The old `MOGRA_API_KEY` name is no longer honored — rename it to `KRAWL_DEV_KEY`.) Key resolution order is:

1. A `krawl_*` key is validated against the multi-user key store (plan-scoped, revocable, expirable).
2. If the key is not a `krawl_*` key, it is compared against `KRAWL_DEV_KEY` (constant-time).
3. If neither is configured, auth is disabled (development only) and the server logs a warning at boot.

The legacy key has no plan, so monthly quota enforcement is a no-op for it. Admin key-management routes deliberately do **not** fall back to `KRAWL_DEV_KEY`. They require a separate `KRAWL_ADMIN_KEY`, so an ordinary data-plane key can never mint quota or read key metadata.

## Billing

Paid plans use Stripe. `POST /billing/checkout` starts a Stripe Checkout session to upgrade a `krawl_live_` key to Pro, and `POST /billing/portal` opens the Stripe billing portal for the key's linked customer. Both require billing to be enabled on the server (`STRIPE_SECRET_KEY` set) and a `krawl_` key in `X-API-Key`. Demo and test keys cannot be upgraded. See the billing endpoints in the [API Reference](/docs/api-reference).

## See Also

- [Payments & x402](/docs/payments-x402): keyless, pay-per-request agent payments in USDC on Base.
- [API Reference](/docs/api-reference): full endpoint, request, and response details, including billing.

## Audit export

Signed-in account owners can download `GET /auth/me/audit/export` using their
account session cookie. An API key does not authorize account-wide audit access.
The version 1 JSON export is paginated (default 100 events, maximum 500). It
contains only events attributable to that account's user ID and current keys.
There is no `user_id` override.

```bash
curl --cookie cookies.txt \
  'https://api.krawl.sh/auth/me/audit/export?limit=100' -o audit.json
```

Pass the returned `next_cursor` as `?cursor=...&limit=100` for the next page.
The cursor preserves the requested time window and position; it is not an
authentication token or an immutable database snapshot. New requests always
check the account session and current ownership. Optional `start` and `end`
parameters must include UTC or another explicit timezone. The maximum window
is `DATA_RETENTION_DAYS` (90 days by default); a read-time cutoff excludes expired
rows even if the purge job has not run yet. An empty history does not prove that
an event never happened.

Events include recorded account/key activity, content-free research usage,
accepted billing-meter submissions, and recorded account-deletion requests.
The latter become inaccessible after account erasure. Billing-meter acceptance
is not an invoice or proof of payment. This export does not claim to include
unrecorded events or every Stripe event.

Each event contains `id`, `at`, `event`, `category`, `actor` and a small `facts`
object. Facts may include a known plan, mode or sign-in method and numeric usage
counts. Query/report text, source URLs and hashes, IP addresses, user agents,
key hashes, raw keys, email addresses, free-text metadata and billing identifiers
are excluded. The response is a private, noncached download; no new export
record is stored. Existing content-free ZDR usage accounting is unchanged.

On successful account erasure, sessions and keys are removed. Any retained
security event loses its user ID, IP, user agent, target and metadata, and a
completed deletion request is labelled `account.erased`. Only content-free
operational history remains for the existing retention period. A caller's
previously downloaded exports remain on the caller's device.

---


# Payments & x402

Source: https://docs.krawl.sh/docs/payments-x402

## What is x402?

x402 turns HTTP `402 Payment Required` into a working payment rail. When an
autonomous agent calls a paid Krawl endpoint without paying, the server answers
with a machine-readable challenge describing the exact price, asset, and
receiving wallet. The agent signs a USDC payment authorization, retries with an
`X-PAYMENT` header, and gets the result.

This matters for agents because it removes every human step:

- **No account, no signup.** There is nothing to register.
- **No API key.** The payment itself is the authorization.
- **No human in the loop.** An agent discovers the price, pays, and gets work
  done in a single retry, all over plain HTTP.

Payments settle in USDC on Base. Krawl never custodies funds or touches a
private key. A facilitator service verifies the signature and broadcasts the
on-chain settlement.

> Krawl implements x402 specification v1 (HTTP transport). The version constant
> surfaced in every response is `x402Version: 1`. Spec:
> [github.com/coinbase/x402](https://github.com/coinbase/x402).

## Pricing

Krawl charges per research request, by tier:

| Tier | Price | Atomic USDC units | Env var |
|---|---|---|---|
| Quick krawl (`mode=quick`) | $1.00 | `1000000` | `X402_PRICE_SHALLOW` |
| Deep krawl (`mode=deep`) | $5.00 | `5000000` | `X402_PRICE_DEEP` |

USDC has 6 decimals, so prices are expressed in atomic units: `1000000` =
$1.00. Both prices are configurable via the Krawl server's environment
variables (see [Server x402 configuration](#server-x402-configuration)). The
amount advertised in the challenge comes through the `maxAmountRequired` field
as a string.

## Which endpoints are gated

x402 gates only the **research endpoints**. Everything else stays on the normal
API-key auth path.

| Endpoint | x402 gated? | Notes |
|---|---|---|
| `POST /research` | Yes | Deep or quick depending on `mode` |
| `POST /research/upload` | Yes | Deep research with uploaded documents |
| `POST /chat/completions` and `POST /v1/chat/completions` | Yes | OpenAI-compatible surface |
| `POST /templates/{template_id}/research` | Yes | Template execution |
| `POST /research/async` | No | Requires an API key (no keyless x402) |
| `POST /search` | No | Requires an API key |
| `GET /results`, `/templates`, `/sessions`, etc. | No | Standard API-key auth only |

> Self-hosting warning: x402 protects the research endpoints, but the other
> endpoints (`/results`, `/templates`, `/sessions`, and friends) use regular
> API-key auth. If you enable x402 without setting `KRAWL_DEV_KEY`, those other
> endpoints stay publicly accessible and the server logs a warning on startup.
> Set `KRAWL_DEV_KEY` to protect them.

## The payment flow

Here is the exact sequence Krawl runs, with the real field names from the
implementation.

### 1. Agent calls a gated endpoint with no payment

The agent POSTs to a research endpoint without an `X-PAYMENT` header (and
without an API key).

```bash
curl -i -N -X POST https://api.krawl.sh/research \
  -H "Content-Type: application/json" \
  -d '{"query": "What is EIP-4844?", "mode": "quick"}'
```

### 2. Server returns HTTP 402 with a payment-requirements challenge

The server responds `402 Payment Required`. The body is rendered at the top
level (not nested under a `detail` key) so standard x402 clients can read
`accepts` and `x402Version` directly. A `WWW-Authenticate: Payment` header is
also emitted for agents that speak the Merchant Payment Protocol.

```
HTTP/1.1 402 Payment Required
Content-Type: application/json
WWW-Authenticate: Payment realm="krawl", methods="x402", discovery="https://api.krawl.sh/discovery/resources"
```

```json
{
  "x402Version": 1,
  "error": "X-PAYMENT header is required",
  "accepts": [
    {
      "scheme": "exact",
      "network": "base",
      "maxAmountRequired": "1000000",
      "asset": "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913",
      "payTo": "0xYourReceivingWalletAddress",
      "resource": "https://api.krawl.sh/research",
      "description": "Krawl quick research (pay-per-use, non-refundable)",
      "mimeType": "text/event-stream",
      "outputSchema": null,
      "maxTimeoutSeconds": 120,
      "extra": {
        "name": "USDC",
        "version": "2"
      }
    }
  ]
}
```

The `accepts` array holds one `PaymentRequirements` object per acceptable
payment. `maxAmountRequired` is the price for the requested tier (here, `1000000`
for a quick krawl). `extra.name` and `extra.version` carry the USDC EIP-712
domain name and version used to construct the signature.

### 3. Agent signs a USDC authorization and retries with `X-PAYMENT`

The agent builds an EIP-3009 transfer authorization that pays at least
`maxAmountRequired` USDC to `payTo`, signs it, and wraps it in a
`PaymentPayload`. That payload is JSON, base64-encoded, and sent in the
`X-PAYMENT` header. The shape Krawl decodes and validates is:

```json
{
  "scheme": "exact",
  "network": "base",
  "payload": {
    "authorization": {
      "to": "0xYourReceivingWalletAddress",
      "value": "1000000"
    }
  }
}
```

```bash
curl -i -N -X POST https://api.krawl.sh/research \
  -H "Content-Type: application/json" \
  -H "X-PAYMENT: eyJzY2hlbWUiOiJleGFjdCIsIm5ldHdvcmsiOiJiYXNlIiwicGF5bG9hZCI6ey4uLn19" \
  -d '{"query": "What is EIP-4844?", "mode": "quick"}'
```

Most agents will use an x402 client library (for example, the Coinbase x402
SDK) to construct and sign this header automatically.

### 4. Server verifies, settles, and streams the cited report

Before trusting the facilitator, Krawl does a defense-in-depth check that the
signed authorization matches the challenge: same `scheme`, same `network`, the
`payload.authorization.to` address equals `payTo`, and the authorized `value` is
greater than or equal to `maxAmountRequired` (overpaying is allowed). A
mismatch returns `402`; a malformed/undecodable `X-PAYMENT` header returns
`400`.

If that passes, Krawl calls the facilitator's `/verify` endpoint, then (after
reserving a concurrency slot) its `/settle` endpoint to broadcast the transfer.
Both calls send `{ "x402Version": 1, "paymentPayload": ..., "paymentRequirements": ... }`.

On success the server streams the cited research report as Server-Sent Events
(`mimeType: text/event-stream`) and attaches an `X-PAYMENT-RESPONSE` header.
That header is a base64-encoded JSON settlement receipt:

```json
{
  "success": true,
  "transaction": "0x…",
  "network": "base",
  "payer": "0x…"
}
```

If settlement fails, the response carries an additional `errorReason` field and
the request is rejected with `402`. If the server is at capacity it returns
`503` with an empty `accepts` array **without charging**, since the slot is
reserved before payment is settled.

## Discovering prices without triggering a 402

Agents can fetch the full payable-resource catalog up front:

```bash
curl https://api.krawl.sh/discovery/resources
```

```json
{
  "x402Version": 1,
  "enabled": true,
  "facilitator": "https://x402.org/facilitator",
  "resources": [
    {
      "resource": "https://api.krawl.sh/research",
      "method": "POST",
      "description": "Deep multi-source research report (cited markdown).",
      "x402": {
        "scheme": "exact",
        "network": "base",
        "asset": "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913",
        "maxAmountRequired": "5000000",
        "priceUsd": 5.0,
        "payTo": "0xYourReceivingWalletAddress"
      }
    }
  ],
  "payment_methods": ["x402", "api_key"]
}
```

## Network and asset details

| Setting | Value | Env var |
|---|---|---|
| Network | `base` (mainnet); `base-sepolia` for testnet | `X402_NETWORK` |
| Asset | USDC | `X402_ASSET_NAME` |
| Asset contract (Base mainnet) | `0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913` | `X402_ASSET` |
| Asset (EIP-712 domain) version | `2` | `X402_ASSET_VERSION` |
| Facilitator | `https://x402.org/facilitator` | `X402_FACILITATOR_URL` |
| Max payment timeout | `120` seconds | `X402_MAX_TIMEOUT_SECONDS` |

The facilitator handles both `/verify` and `/settle`. The verify/settle timeout
is aligned with `maxTimeoutSeconds` so a Base settlement broadcast has room to
confirm under congestion without a premature client timeout.

## Server x402 configuration

x402 turns on automatically once a receiving wallet address is set on the
Krawl server. This section is operator reference — it documents how the
Krawl x402 surface is configured. Hosted Krawl
(`api.krawl.sh`) ships with these defaults; the env vars are listed here so
the contract between client and server is explicit.

```bash
# Enable x402 by setting your receiving wallet (0x… on Base)
X402_PAY_TO=0xYourReceivingWalletAddress

# Choose the network: "base" (mainnet) or "base-sepolia" (testnet)
X402_NETWORK=base

# Prices in atomic USDC units (6 decimals). Defaults shown.
X402_PRICE_SHALLOW=1000000   # $1.00 per quick krawl
X402_PRICE_DEEP=5000000      # $5.00 per deep krawl

# Optional: protect non-research endpoints when x402 is on
KRAWL_DEV_KEY=your-shared-key
```

For the full list of environment variables, see
[Getting Started](/docs/getting-started).

## x402 and API keys are alternatives

A request authenticated with a valid API key is **not** charged via x402. The
research endpoints accept either form of access:

- **API key present** (`X-API-Key` header): the request is authorized by the
  key and billed against the key's plan/quota. x402 does not run.
- **No API key, x402 enabled**: the request must carry a valid `X-PAYMENT`
  header and pays per request in USDC.

You never pay twice. If you hold a key, just send it. See
[Authentication](/docs/authentication) for keys, plans, and quotas, and
[Agents & MCP](/docs/agents-mcp) for connecting autonomous agents to Krawl.

## Quickstart — runnable code

### Trigger a live 402 challenge (curl)

The fastest way to confirm x402 is alive: call a gated endpoint with no key
and no `X-PAYMENT`. The response body is the challenge.

```bash
curl -i -X POST https://api.krawl.sh/research \
  -H "Content-Type: application/json" \
  -d '{"query": "What is EIP-4844?", "mode": "quick"}'
```

You will see `HTTP/1.1 402 Payment Required` and a JSON body with `accepts[0].maxAmountRequired = "1000000"` (one quick krawl, $1.00).

### Python — sign + pay with the x402 SDK

```python
# pip install x402 web3 httpx
import os, httpx
from eth_account import Account
from x402.client import X402Client

pk = os.environ["AGENT_PRIVATE_KEY"]              # 0x… Base wallet
account = Account.from_key(pk)
client = X402Client(account=account, http=httpx.Client(timeout=600))

resp = client.post(
    "https://api.krawl.sh/research",
    json={"query": "What is EIP-4844?", "mode": "quick"},
    headers={"Accept": "text/event-stream"},
)
# x402 SDK auto-handled the 402 challenge, signed an EIP-3009 transfer,
# replayed with X-PAYMENT, and the server is now streaming SSE.
for line in resp.iter_lines():
    if line.startswith("data: "):
        print(line)
print("settled:", resp.headers.get("X-PAYMENT-RESPONSE"))
```

### Node — sign + pay with the x402 SDK

```ts
// npm i x402 viem
import { createX402Client } from "x402";
import { privateKeyToAccount } from "viem/accounts";

const account = privateKeyToAccount(process.env.AGENT_PRIVATE_KEY as `0x${string}`);
const client = createX402Client({ account });

const resp = await client.fetch("https://api.krawl.sh/research", {
  method: "POST",
  headers: { "Content-Type": "application/json", Accept: "text/event-stream" },
  body: JSON.stringify({ query: "What is EIP-4844?", mode: "quick" }),
});

// X-PAYMENT was attached, server settled, body is the SSE stream
const reader = resp.body!.getReader();
const dec = new TextDecoder();
while (true) {
  const { done, value } = await reader.read();
  if (done) break;
  process.stdout.write(dec.decode(value, { stream: true }));
}
console.error("settled:", resp.headers.get("X-PAYMENT-RESPONSE"));
```

## Testnet — Base Sepolia

Point the same code at Base Sepolia by setting the network on your x402
client (or, in a local dev environment, on the server). The challenge will
advertise `network: "base-sepolia"` and Sepolia USDC, and the facilitator
will verify against the testnet:

```bash
# Local dev: enable testnet on the server.
X402_NETWORK=base-sepolia
X402_ASSET=0x036CbD53842c5426634e7929541eC2318f3dCF7e   # USDC on Base Sepolia
```

```python
# Python x402 client — explicit network selection
X402Client(account=account, network="base-sepolia")
```

```ts
// Node x402 client — explicit network selection
createX402Client({ account, network: "base-sepolia" })
```

Funded Sepolia USDC is available from the
[Circle USDC faucet](https://faucet.circle.com).

## Verify settlement after the call

Every successful x402 response carries `X-PAYMENT-RESPONSE` (base64 JSON):

```bash
echo "<X-PAYMENT-RESPONSE value>" | base64 -d | jq
```

```json
{
  "success": true,
  "transaction": "0xabc...",
  "network": "base",
  "payer": "0x..."
}
```

Use `transaction` to look the transfer up on
[BaseScan](https://basescan.org) (`https://sepolia.basescan.org` for
testnet). If the facilitator's `success` flag is `false`, an `errorReason`
field explains why and the request was rejected with `402`.

---


# Agents & MCP

Source: https://docs.krawl.sh/docs/agents-mcp

Krawl is built to be driven by autonomous agents, not just humans. There are three ways an agent can call it, plus a full discovery surface so an agent can find and pay for it on its own.

- **MCP server** — connect Krawl as a Model Context Protocol tool (Claude, Cursor, ChatGPT, Smithery, mcp.so).
- **OpenAI-compatible shim** — point any OpenAI SDK at Krawl and get a cited report back.
- **Native HTTP + SSE** — the [research API](/docs/api-reference) directly.

Authentication on all three is the same: an [API key](/docs/authentication), a no-signup demo key, or keyless [x402](/docs/payments-x402) payments.

## MCP Server

Krawl exposes a Model Context Protocol server at `POST /mcp` using the **Streamable HTTP** transport (JSON-RPC 2.0 over a single POST).

- **Default protocol version:** `2025-06-18`
- **Server:** `krawl` v`0.2.0`
- **Methods:** `initialize`, `ping`, `tools/list`, `tools/call`

Discovery methods (`initialize`, `ping`, `tools/list`) are **never gated**, so an agent platform can connect and enumerate tools without credentials. Only `tools/call` execution is authenticated, mirroring the research endpoints (`X-API-Key`, or keyless x402 when enabled).

### Tools

Each tool advertises its input schema and (where applicable) an x402
`price` (USDC atomic units) on `tools/list`. Per-tool x402 means an agent
can shop the catalog before signing a payment.

| Tool | x402 price | Schema (highlights) | What it does |
|---|---|---|---|
| `research` | `5000000` ($5.00) | `query: string` (req); `mode: enum`; `breadth: int`; `depth_levels: int`; `sources: string[]`; `output_schema?: object` | Full recursive deep-research run, cited markdown |
| `quick_search` | `1000000` ($1.00) | `query: string` (req); `sources?: string[]`; `limit?: int` | Flat single-shot multi-source search |
| `ask_krawl` | `1000000` ($1.00) | `question: string` (req); `mode?: "deep"\|"quick"` | NLWeb-style natural-language ask |
| `list_lookouts` | free | none | Enumerate scheduled lookouts |
| `get_lookout` | free | `lookout_id: string` (req) | Fetch a lookout |
| `run_lookout` | `5000000` ($5.00) | `lookout_id: string` (req) | Trigger a lookout run now |
| `export_result` | free | `result_id: string` (req); `format: enum` | Export a result (pdf/html/md/docx/slides/run-record JSON) |
| `get_usage` | free | none | Current quota and spend |
| `get_memory` | free | `query?: string`; `limit?: int` | Read cross-session knowledge base |
| `compare_results` | `1000000` ($1.00) | `a_id: string` (req); `b_id: string` (req) | Diff two finished research results |

Free tools are read-only; paid tools advertise `price` in the
`tools/list` response so an agent can budget. Without a key, `tools/call`
on a paid tool returns 402 with an x402 challenge.

### Protocol compatibility

Legacy clients continue to call `initialize`. It returns `2025-06-18`,
including when the client requests a later version through that handshake.
GET discovery and the existing legacy SSE endpoint remain available.

An experimental `2026-07-28` transport is implemented behind
`MCP_MODERN_ENABLED=false`. Keep it disabled in deployments until the
modern authorization and reference-client acceptance in issue #120 pass.
When enabled, each POST carries its version, client identity and capabilities
in `params._meta`; no initialization or session identifier is required.
`server/discover` advertises the enabled versions. Supply
`MCP-Protocol-Version`, `Mcp-Method` and, for named operations,
`Mcp-Name` matching the body. Accept both JSON and event streams.
Modern successful results include `resultType: "complete"`; discovery,
lists and resource reads include cache hints. Private resource reads are
scoped to the authenticated caller and marked private with zero TTL.

The experimental transport reuses Krawl's existing API-key/x402 authorization,
quota and metering. This is not a completed OAuth 2.1 migration: resource
indicators, issuer/audience validation, modern 401/403 challenges, client
metadata registration and live Registry installation still need acceptance.
Krawl does not advertise support for subscriptions, Tasks or multi-round-trip
requests. The [MCP specification](https://modelcontextprotocol.io/specification/2026-07-28)
defines those separate capabilities.

### Install in Claude Code

The skill installs research instructions and a bundled shell client. Configure
MCP separately in the next section. Repository installation currently requires
GitHub read access because the source repository is private.

```bash
npx skills@1.7.4 add ravidsrk/krawl --skill krawl --agent claude-code
```

From an existing checkout, use `npx skills@1.7.4 add ./skills/krawl --skill krawl --agent claude-code --copy --yes`.
The canonical [skill directory](https://github.com/ravidsrk/krawl/tree/main/skills/krawl)
includes its scripts and reference files. Public distribution remains a release
step; packaging alone does not configure MCP or validate a live query.

### Connect

Add Krawl as an MCP server in your agent platform:

```json
{
  "mcpServers": {
    "krawl": {
      "url": "https://api.krawl.sh/mcp",
      "headers": { "X-API-Key": "krawl_live_…" }
    }
  }
}
```

A `GET /mcp` returns a friendly capability pointer (server name, version, protocol, transport, and tool names) for probes that hit it with the wrong method.

### Raw JSON-RPC

```bash
# List tools (no credentials required)
curl -X POST https://api.krawl.sh/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc":"2.0","id":1,"method":"tools/list"}'

# Call a tool (authenticated, or keyless via x402)
curl -X POST https://api.krawl.sh/mcp \
  -H "Content-Type: application/json" \
  -H "X-API-Key: krawl_live_…" \
  -d '{
    "jsonrpc":"2.0","id":2,"method":"tools/call",
    "params":{"name":"quick_search","arguments":{"query":"What is EIP-4844?"}}
  }'
```

A `tools/call` drains the research SSE stream to completion and returns a single text result.

## OpenAI-Compatible Shim

Point any OpenAI client (or LangChain, LlamaIndex, etc.) at Krawl by setting `base_url`. The `model` selects the [research mode](/docs/research-modes); the last user message is the research query. The assistant message is a fully-cited report. Both streaming and non-streaming are supported.

Endpoints: `POST /v1/chat/completions` (and `/chat/completions`), `GET /v1/models` (and `/models`).

### Models

Each research mode maps to a `krawl-*` model id:

| Model id | Mode |
|---|---|
| `krawl-deep` | deep (default) |
| `krawl-quick` | quick |
| `krawl-crypto` | crypto |
| `krawl-token-analysis` | token-analysis |
| `krawl-protocol-research` | protocol-research |
| `krawl-whale-tracking` | whale-tracking |
| `krawl-narrative` | narrative |
| `krawl-risk-assessment` | risk-assessment |
| `krawl-yield-strategy` | yield-strategy |

### Example (Python)

```python
from openai import OpenAI

client = OpenAI(
    base_url="https://api.krawl.sh/v1",
    api_key="krawl_live_…",
)

resp = client.chat.completions.create(
    model="krawl-deep",
    messages=[{"role": "user", "content": "What is the state of AI agents?"}],
)
print(resp.choices[0].message.content)  # cited markdown report
```

### Example (curl)

```bash
curl -X POST https://api.krawl.sh/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-API-Key: krawl_live_…" \
  -d '{
    "model": "krawl-quick",
    "messages": [{"role": "user", "content": "What is EIP-4844?"}],
    "stream": false
  }'
```

When a request is paid via x402, the settlement receipt is returned in the `X-PAYMENT-RESPONSE` header. See [Payments & x402](/docs/payments-x402).

## Discovery

Krawl publishes machine-readable descriptors so an agent can find, understand, and pay for it without a human.

| Resource | What it is |
|---|---|
| `GET /discovery/resources` | Agent-payable resources with per-request price, network, and asset (x402 / MPP convention) |
| `https://krawl.sh/agent.json` | Full agent manifest: endpoints, plans, quotas, modes, providers, MCP, payment protocols, SDKs |
| `https://krawl.sh/skill.md` | Drop-in skill file for Claude Code and other agent runtimes |
| `https://krawl.sh/.well-known/agent-card.json` | A2A-style agent card |
| `https://krawl.sh/.well-known/mcp/manifest.json` | MCP manifest and server card |
| `https://krawl.sh/.well-known/ai-plugin.json` | OpenAI plugin descriptor |
| `https://krawl.sh/.well-known/llms.txt` | LLM-readable site summary |
| `https://krawl.sh/.well-known/oauth-protected-resource` | OAuth protected-resource metadata |

### Resource discovery

`GET /discovery/resources` lets an agent learn what costs what before triggering a 402:

```json
{
  "x402Version": 1,
  "enabled": true,
  "facilitator": "https://x402.org/facilitator",
  "resources": [
    {
      "resource": "https://api.krawl.sh/research",
      "method": "POST",
      "description": "Deep multi-source research report (cited markdown).",
      "x402": {
        "scheme": "exact",
        "network": "base",
        "asset": "0x833589fCD6eDb6E08f4c7C32D4f71b54bdA02913",
        "maxAmountRequired": "5000000",
        "priceUsd": 5.0,
        "payTo": "0x…"
      }
    }
  ],
  "payment_methods": ["x402", "api_key"],
  "auth_docs": "https://krawl.sh/auth.md"
}
```

## Next Steps

- [Authentication](/docs/authentication) — API keys, demo keys, plans, and quotas
- [Payments & x402](/docs/payments-x402) — keyless per-request agent payments
- [API Reference](/docs/api-reference) — every endpoint and schema
- [Research Modes](/docs/research-modes) — what each `krawl-*` model does

---


# CLI

Source: https://docs.krawl.sh/docs/cli

The `krawl` CLI wraps the [REST API](/docs/api-reference) so you can run
research, manage async tasks, and configure lookouts from a terminal or shell
script.

> **Status: documented contract.** This page is the specification for the CLI's
> commands and behavior. The published package is rolling out; until it lands on
> the public registries, this is the canonical contract the CLI implements.
> (Actual package publishing is tracked separately and out of scope here.)

## Install

The CLI is distributed as a Python package (primary) with a Node wrapper:

```bash
# Python (recommended — isolated install)
pipx install krawl-cli
# or
pip install krawl-cli

# Node
npm install -g @krawl/cli
```

Verify:

```bash
krawl --version
```

## Authentication

The CLI reads your API key from the `KRAWL_API_KEY` environment variable:

```bash
export KRAWL_API_KEY="krawl_live_xxxxxxxxxxxxxxxxxxxxxxxx"
```

You can override the base URL (e.g. to point at a staging or proxied Krawl
deployment) with `KRAWL_BASE_URL` (default `https://api.krawl.sh`). A
`--api-key` flag overrides the env var per command. The CLI sends the key in
the `X-API-Key` header on every request, just like the REST API. See
[Authentication](/docs/authentication).

## Common commands

### Run research

```bash
# Stream a deep research run (SSE) to your terminal
krawl research "What is EIP-4844 and who benefits?"

# Pick a mode and tune intensity
krawl research "State of L2 rollups" --mode deep --depth 80 --breadth 5

# Quick mode for fast answers
krawl research "Latest Solana TPS" --mode quick
```

Output defaults to rendered markdown. Use `--json` for the raw result object or
`--output report.md` to write the report to a file.

### Async tasks

```bash
# Submit a background job; prints the task id
krawl research "Long multi-source report" --async

# List your tasks
krawl tasks list

# Check / wait on a task
krawl tasks get <task_id>
krawl tasks wait <task_id>        # blocks until done|error, then prints result
```

### Compare results

```bash
krawl compare "AI agents" --result-ids 9f1c... 7b3e...
```

Maps to [`POST /research/compare`](/docs/batch#pattern-b--post-researchcompare).

### Lookouts

```bash
krawl lookouts create "Watch AI agent funding" --schedule daily \
  --webhook-url https://example.com/notify
krawl lookouts list
krawl lookouts run <lookout_id>
krawl lookouts delete <lookout_id>
```

See [Lookouts](/docs/lookouts) and [Webhooks](/docs/webhooks).

## Global flags

| Flag | Description |
|------|-------------|
| `--api-key <key>` | Override `KRAWL_API_KEY` for this command |
| `--base-url <url>` | Override `KRAWL_BASE_URL` (default `https://api.krawl.sh`) |
| `--json` | Emit raw JSON instead of rendered output |
| `--output <file>` | Write output to a file |
| `--quiet` | Suppress progress/log lines |

## Exit codes

The CLI maps API failures to non-zero exit codes so scripts can branch on them:

| Code | Meaning |
|------|---------|
| `0` | Success |
| `1` | Generic / unexpected error |
| `2` | Usage error (bad flags/args) |
| `3` | Auth error (`401`) |
| `4` | Rate limited (`429`) — honor `Retry-After` and retry |
| `5` | Quota exhausted (`402` `quota_exceeded`) |

On `429`/`5xx` the CLI retries with exponential backoff + jitter, honoring
`Retry-After`. See [Rate Limits & Retries](/docs/rate-limits).

---


# Architecture

Source: https://docs.krawl.sh/docs/architecture

## High-Level Flow

```
Client POST /research { query, mode, breadth, depth_levels }
│
├─ Phase 1: Planning
│    Model: Sonnet 4.6
│    Output: ResearchPlan { topics[]: { title, tasks[] } }
│
├─ Phase 1.5: STORM Perspective Discovery
│    Model: Haiku 4.5
│    Output: 3-5 perspectives with key questions
│    (Skipped for short/simple queries < 30 chars)
│
├─ Phase 2: Recursive Tree Search
│    For each depth level (default 3):
│      ├─ Generate N queries (Haiku 4.5, N = breadth)
│      ├─ Execute all N in parallel via tool executors
│      ├─ Extract learnings (Sonnet 4.6)
│      ├─ Deduplicate learnings (Jaccard similarity ≥ 0.7)
│      ├─ Detect knowledge gaps (Sonnet 4.6)
│      ├─ Update dynamic outline
│      └─ If depth > 0 and gaps exist → recurse (depth-1, breadth/2)
│
├─ Phase 3: Verified Synthesis (Opus 4.7)
│    ├─ Build source block (max 25 sources, 1500 chars each)
│    ├─ Streaming synthesis with retrieve-then-cite verification
│    ├─ Gap check → optional re-synthesis
│    └─ Multi-pass review → revise (up to 2 passes, threshold 0.8)
│
├─ Post-processing
│    ├─ Data visualization extraction
│    ├─ Conflict detection
│    ├─ Structured output extraction (if output_schema provided)
│    └─ Memorize key learnings to knowledge base
│
└─ SSE stream throughout all phases
```

## Mode Dispatch

Krawl has two execution paths:

- **quick** → flat agent loop (LLM picks tools iteratively, `tool_choice='required'`)
- **deep / crypto / all others** → recursive breadth×depth tree search

## Recursive Deep Research

The core engine for `deep` mode:

1. **Query Generation** — Haiku generates N search queries per depth level, informed by accumulated learnings and STORM perspectives
2. **Parallel Execution** — all N queries execute concurrently across available tools
3. **Learning Extraction** — Sonnet extracts structured learnings from search results with confidence scores
4. **Deduplication** — Jaccard word overlap with stemming (threshold 0.7) prevents redundant learnings
5. **Gap Detection** — Sonnet identifies remaining knowledge gaps with priority scores and suggested tools
6. **Outline Update** — dynamic outline evolves as evidence accumulates
7. **Recursion** — if gaps remain and depth > 0, recurse with `depth-1` and `breadth/2`

### Configuration

| Parameter | Default | Range | Description |
|---|---|---|---|
| `breadth` | 4 | 1–10 | Parallel queries per depth level |
| `depth_levels` | 3 | 1–5 | Recursion depth levels |
| `depth` | 75 | 1–100 | Research depth intensity (affects synthesis detail) |

## STORM Perspective Discovery

Inspired by [Stanford's STORM](https://arxiv.org/abs/2402.14207), Krawl discovers 3–5 research perspectives before starting the search tree. Each perspective has:

- **name** — the perspective role (e.g., "Financial Analyst", "Security Researcher")
- **description** — what this perspective focuses on
- **key_questions** — specific questions this perspective would ask

Perspectives are injected into query generation, ensuring the research tree covers multiple angles rather than a single viewpoint. Skipped for short queries (< 30 chars or single word).

## Multi-Pass Synthesis

Synthesis happens in 3 sub-phases:

### Initial Synthesis (Opus 4.7)

One streaming LLM call generates the full report from learnings + source content. Uses `retrieve-then-cite` — only sources whose content was actually fetched can be cited.

### Gap Check (Sonnet 4.6)

After initial synthesis, a quality gate reviews the report for missing coverage. If gaps are found, a re-synthesis pass incorporates the missing aspects.

### Review → Revise Loop (up to 2 passes)

A review model scores the report (0–1) and lists issues. If the score is below 0.8, a revision pass is triggered. The loop runs at most 2 times (configurable via `MAX_REVISION_PASSES`).

```
Initial synthesis (Opus 4.7)
    └─ Gap check (Sonnet 4.6)
        └─ If gaps found → Re-synthesize (Opus 4.7)
    └─ Review pass 1 (Sonnet 4.6) → score < 0.8?
        └─ Revise pass 1 (Opus 4.7)
    └─ Review pass 2 (Sonnet 4.6) → score < 0.8?
        └─ Revise pass 2 (Opus 4.7)
    └─ Accept report
```

## Flat Agent Loop (Quick Mode)

For `quick` mode, the original agent loop is used:

1. **Planning** — structured research plan
2. **Agent loop** — LLM iteratively picks tools with `tool_choice='required'`, max 40 steps
3. **Synthesis** — single-pass report generation
4. **Message trimming** — when conversation exceeds 44 messages, trim to keep first 2 + last 40

## Agent Tools

The LLM has access to these tools during research:

| Tool | Source Type | Description |
|---|---|---|
| `thinking` | — | Mandatory reflection between tool calls |
| `webSearch` | exa, web | Search via Exa (up to 5 parallel queries) |
| `browsePage` | exa, web | Read full content from URLs (up to 5 URLs) |
| `firecrawlSearch` | web | JS-heavy site search with full content extraction |
| `newsSearch` | web | Recent news articles via Firecrawl |
| `xSearch` | x | X/Twitter search for sentiment and breaking news |
| `xTimeline` | x | Fetch a specific user's recent tweets |
| `xCashtagSearch` | x | Search by cashtag ($BTC, $ETH) |
| `githubSearch` | github | Search GitHub repositories |
| `codeSearch` | github | Search code across repositories |
| `cryptoData` | crypto | Token price, market cap, supply from CoinGecko |
| `cryptoChart` | crypto | Price history and chart data |
| `defiRanking` | crypto | Protocol TVL rankings from DeFi Llama |
| `onchainAnalytics` | crypto | On-chain metrics (active addresses, tx count) |
| `protocolDeepDive` | crypto | Deep protocol analysis |
| `tokenRiskAssessment` | crypto | Security and risk analysis |
| `smartMoneyTracker` | crypto | Whale and institutional tracking |
| `narrativeTracker` | crypto | Social sentiment and narrative analysis |
| `derivativesAnalysis` | crypto | Funding rates, open interest, liquidations |
| `yieldFinder` | crypto | APY/APR and yield opportunities |
| `governanceMonitor` | crypto | DAO proposals and governance activity |
| `academicSearch` | academic | Semantic Scholar papers |
| `pubmedSearch` | academic | PubMed biomedical research |
| `mediaSearch` | media | YouTube search with transcription |
| `redditSearch` | community | Reddit discussions |
| `hackerNewsSearch` | community | Hacker News threads |
| `wikipediaSearch` | community | Wikipedia articles |
| `tavilySearch` | web | Tavily search (if API key set) |
| `serperSearch` | web | Serper/Google search (if API key set) |
| `braveSearch` | web | Brave search (if API key set) |
| `duckduckgoSearch` | web | DuckDuckGo search (no API key needed) |
| `done` | — | Signal search phase completion |

### Tool Filtering

The `sources` parameter in the research request controls which tools are available:

```json
{
  "query": "...",
  "sources": ["exa", "x", "crypto"]
}
```

Only tools mapped to the specified source types are exposed to the LLM. The `thinking` and `done` tools are always available.

## Concurrency

- **Research semaphore**: max 3 concurrent research sessions
- **Parallel searches**: all queries at each depth level run concurrently
- **Rate limiting**: 5 requests/minute per IP (configurable)

## Security

- API key auth via `X-API-Key` header (HMAC comparison)
- Security headers: HSTS, X-Frame-Options DENY, X-Content-Type-Options nosniff
- Request body limits: 1MB (normal), 50MB (file uploads)
- SSRF prevention on webhook URLs
- Global exception handler sanitizes all errors

---


# Model Strategy

Source: https://docs.krawl.sh/docs/models

## 4-Tier Model Strategy

Krawl uses a tiered model approach — cheaper/faster models for high-volume tasks, premium models for intelligence-critical work.

| Tier | Model | Tasks | Call Volume |
|---|---|---|---|
| **Tier 1** (budget) | `bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0` | Query generation, URL ranking, perspective discovery, memory extraction | High (10–30+ per session) |
| **Tier 2** (mid) | `bedrock/us.anthropic.claude-sonnet-4-6` | Learning extraction, flat agent loop | Moderate (5–15 per session) |
| **Tier 3** (quality) | `bedrock/us.anthropic.claude-sonnet-4-6` | Planning, gap analysis, review passes | Low (2–5 per session) |
| **Tier 4** (apex) | `bedrock/us.anthropic.claude-opus-4-7` | Synthesis, revision passes | 1 call per session |
| **xAI** | `grok-4.20-0309-non-reasoning` | X/Twitter search only | Per X search |

### Why This Approach

- **Cost control** — Haiku handles 60%+ of LLM calls at ~10x lower cost than Opus
- **Quality where it matters** — Opus only fires once for the final synthesis, producing the best possible report
- **Speed** — Haiku is fast enough for real-time query generation during recursive search

## Provider Routing

### Primary: AWS Bedrock

All Claude models route through AWS Bedrock via litellm. The `us.` prefix uses **cross-region inference profiles** — AWS automatically routes to the least loaded region.

```
bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0
bedrock/us.anthropic.claude-sonnet-4-6
bedrock/us.anthropic.claude-opus-4-7
```

Required env vars:
- `AWS_BEDROCK_ACCESS_KEY_ID`
- `AWS_BEDROCK_SECRET_ACCESS_KEY`
- `AWS_BEDROCK_REGION` (default: `us-east-1`)

### Fallback: Anthropic Direct API

If any Bedrock call fails, Krawl automatically retries with the equivalent Anthropic direct API model:

| Bedrock Model | Fallback Model |
|---|---|
| `bedrock/us.anthropic.claude-sonnet-4-6` | `anthropic/claude-sonnet-4-6` |
| `bedrock/us.anthropic.claude-opus-4-7` | `anthropic/claude-opus-4-7` |
| `bedrock/us.anthropic.claude-haiku-4-5-20251001-v1:0` | `anthropic/claude-haiku-4-5-20251001` |

Requires `ANTHROPIC_API_KEY` to be set.

### Cross-Provider Fallback

The Bedrock and Anthropic models are the same Claude family, so a full Anthropic outage would take out every tier. To survive that, Krawl maps each Claude model to an **independent provider** (OpenAI by default):

| Claude Model | Cross-Provider Fallback |
|---|---|
| `claude-opus-4-7` (synthesis) | `openai/gpt-5` |
| `claude-sonnet-4-6` (planning/research) | `openai/gpt-5-mini` |
| `claude-haiku-4-5` (budget) | `openai/gpt-5-nano` |

Requires the relevant provider key (`OPENAI_API_KEY`). The map is configurable via `FALLBACK_MODEL_MAP`; cross-provider fallback is disabled if the map is empty.

### OpenRouter Budget Override (Optional)

For cost optimization, the high-volume Tier-1 budget calls (query generation, URL ranking, outlines — anything using `MODEL_QUERY_GEN`) can be routed through a free OpenRouter model. A research session makes 20-30 budget-tier calls, so this is where the savings are. Enable it by setting `OPENROUTER_API_KEY` and the OpenRouter override. OpenRouter `:free` models meter as `$0`.

### xAI (X/Twitter Search)

X/Twitter search uses xAI's Grok model directly — **not** through litellm or Bedrock. The API call goes straight to `https://api.x.ai/v1`.

Model: `grok-4.20-0309-non-reasoning`

Requires `XAI_API_KEY`.

## Circuit Breaker

Krawl implements a circuit breaker pattern for Bedrock:

1. Each Bedrock failure increments a counter
2. After **3 consecutive failures**, the circuit opens
3. While open (5 minutes), all calls go **directly to Anthropic fallback** — skipping Bedrock entirely
4. On the next successful Bedrock call, the circuit resets

```
Normal: Bedrock → (success) → reset counter
         Bedrock → (fail) → counter++ → try Anthropic fallback
         Bedrock → (fail) → counter++ → try Anthropic fallback
         Bedrock → (fail) → counter=3 → CIRCUIT OPEN

Open:   Skip Bedrock → Anthropic direct (for 5 minutes)

After 5min: Try Bedrock again
         Bedrock → (success) → CIRCUIT RESET → counter=0
```

Constants:
- `_CIRCUIT_FAIL_THRESHOLD = 3`
- `_CIRCUIT_OPEN_SECONDS = 300` (5 minutes)

## Model Override

You can override any model at the request level:

```json
{
  "query": "...",
  "model": "anthropic/claude-sonnet-4-6"
}
```

This overrides **all phases** (planning, research, synthesis). Useful for testing or when you want to force a specific provider.

You can also override via env vars:

```bash
MODEL_PLANNING=anthropic/claude-sonnet-4-6
MODEL_RESEARCH=anthropic/claude-sonnet-4-6
MODEL_SYNTHESIS=anthropic/claude-opus-4-7
MODEL_QUERY_GEN=anthropic/claude-haiku-4-5-20251001
```

## Opus 4.7 Notes

- Opus 4.7 **does not support** the `temperature` parameter — Krawl automatically omits it
- Used only for synthesis (1 call per session) due to cost
- Produces the highest quality reports with better citation handling

---


# Research Modes

Source: https://docs.krawl.sh/docs/research-modes

Krawl supports 9 research modes. The mode determines which tools are prioritized, how the query is processed, and what the report focuses on.

## Modes Overview

| Mode | Engine | Best For |
|---|---|---|
| `deep` | Recursive tree search | General research, comprehensive analysis |
| `quick` | Flat agent loop | Fast overviews, simple questions |
| `crypto` | Recursive tree search | General crypto/DeFi research |
| `token-analysis` | Recursive tree search | Single token deep dive |
| `protocol-research` | Recursive tree search | DeFi protocol analysis |
| `whale-tracking` | Recursive tree search | Smart money and institutional flows |
| `narrative` | Recursive tree search | Market narratives and sentiment |
| `risk-assessment` | Recursive tree search | Security and risk analysis |
| `yield-strategy` | Recursive tree search | Yield farming and APY optimization |

## `deep`

The default mode. Full recursive breadth×depth tree search with STORM perspectives, learning extraction, gap detection, and multi-pass synthesis.

```bash
curl -N -X POST https://api.krawl.sh/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{
    "query": "What is the current state of quantum computing?",
    "mode": "deep",
    "breadth": 4,
    "depth_levels": 3
  }'
```

## `quick`

Uses the flat agent loop instead of recursive tree search. The LLM iteratively picks tools (`tool_choice='required'`) up to `MAX_STEPS` (40). Faster but less thorough.

```bash
curl -N -X POST https://api.krawl.sh/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{
    "query": "What is MCP protocol?",
    "mode": "quick"
  }'
```

## `crypto`

General crypto research mode. Automatically includes crypto tools (cryptoData, cryptoChart, defiRanking, etc.) alongside standard web and X/Twitter search.

```bash
curl -N -X POST https://api.krawl.sh/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{
    "query": "What is happening in the Solana ecosystem?",
    "mode": "crypto"
  }'
```

## `token-analysis`

Deep dive on a single token. Prioritizes cryptoData, cryptoChart, tokenRiskAssessment, smartMoneyTracker, and X/Twitter sentiment.

```bash
curl -N -X POST https://api.krawl.sh/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{
    "query": "Comprehensive analysis of Ethereum",
    "mode": "token-analysis"
  }'
```

## `protocol-research`

Focused on DeFi protocol analysis. Prioritizes defiRanking, protocolDeepDive, governanceMonitor, and smart contract code review.

```bash
curl -N -X POST https://api.krawl.sh/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{
    "query": "Analyze Aave v3 protocol mechanics and governance",
    "mode": "protocol-research"
  }'
```

## `whale-tracking`

Tracks smart money and institutional flows. Prioritizes smartMoneyTracker, onchainAnalytics, and derivativesAnalysis.

```bash
curl -N -X POST https://api.krawl.sh/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{
    "query": "Where is smart money flowing in DeFi?",
    "mode": "whale-tracking"
  }'
```

## `narrative`

Analyzes market narratives and social sentiment. Prioritizes narrativeTracker, X/Twitter search, and community sources (Reddit, Hacker News).

```bash
curl -N -X POST https://api.krawl.sh/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{
    "query": "What is the AI x Crypto narrative?",
    "mode": "narrative"
  }'
```

## `risk-assessment`

Security and risk analysis. Prioritizes tokenRiskAssessment, audit history, exploit research, and smart contract vulnerability scanning.

```bash
curl -N -X POST https://api.krawl.sh/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{
    "query": "Risk assessment of bridging protocols",
    "mode": "risk-assessment"
  }'
```

## `yield-strategy`

Yield farming and APY optimization research. Prioritizes yieldFinder, defiRanking, and protocol analysis tools.

```bash
curl -N -X POST https://api.krawl.sh/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{
    "query": "Best stablecoin yield strategies in 2025",
    "mode": "yield-strategy"
  }'
```

## Source Type Filtering

Regardless of mode, you can control which source types are available:

```json
{
  "query": "...",
  "mode": "deep",
  "sources": ["exa", "x", "academic"]
}
```

Available source types: `exa`, `x`, `github`, `crypto`, `web`, `academic`, `media`, `community`

Each source type maps to specific tools. See [Architecture](/docs/architecture) for the full tool-to-source mapping.

---


# API Reference

Source: https://docs.krawl.sh/docs/api-reference

Base URL: `https://api.krawl.sh`

Version: `0.2.0`

## Authentication

Most endpoints accept an optional API key via the `X-API-Key` header. Behavior depends on how the server is configured:

- `krawl_*` keys are validated against the keys table (multi-user, plan-scoped, revocable, expirable).
- A legacy single `KRAWL_DEV_KEY` is honored for backward compatibility.
- If neither is configured, auth is disabled (development only).

```bash
curl -H "X-API-Key: your-key" https://api.krawl.sh/health
```

When auth is enabled and the key is wrong, missing, revoked, or expired:

```json
{"detail": "Invalid, revoked, or expired API key"}
```

The research endpoints (`/research`, `/research/upload`, `/templates/{id}/research`, `/v1/chat/completions`) also support keyless pay-per-request via x402 when enabled. Admin-only endpoints require a separate `KRAWL_ADMIN_KEY`. See [Authentication](/docs/authentication) for the full model.

## Rate limits, retries & status

`Idempotency-Key` replays eligible successful JSON responses for up to 24 hours.
Authentication and key-management routes under `/auth`, token issuance at
`/agent/auth`, requests carrying the
session cookie or an `Authorization` header, legacy or unknown `X-API-Key`
credentials, ZDR requests, ephemeral evidence
exports, SSE, `no-store` and `Set-Cookie` responses bypass this response cache.
Authentication always runs again for these requests; a retry of key issuance
can create a new key. Do not rely on this header to recover a lost credential.
Previously cached responses are invalidated when this security update is deployed.

Rate limit: **5 requests/minute** per IP by default (some endpoints set their own
limits, noted below). Every response carries `X-RateLimit-Limit` /
`X-RateLimit-Remaining` / `RateLimit-Policy` headers so you can self-throttle.

When you exceed a limit the API returns **`429 Too Many Requests`** with a
**`Retry-After`** header (delta-seconds) and a structured envelope
(`{ "error_code": "RATE_LIMITED", "retry_after": 60, ... }`). Agents **must**
honor `Retry-After` and apply exponential backoff + jitter on `429`/`5xx`. See
the full [Rate Limits & Retries](/docs/rate-limits) playbook for thresholds, the
429 envelope, and a concrete backoff loop.

- **Batch/bulk:** [`POST /research/batch`](/docs/batch) queues up to 20 jobs in
  one call.
- **Status:** live [status page](/status) and [`/status.json`](/status.json);
  programmatic [`GET /health`](https://api.krawl.sh/health). Use these to
  determine retry eligibility and recover from failure streaks
  ([error-recovery playbook](/docs/status#error-recovery-playbook)).

---

## Health

### `GET /health`

Liveness and readiness check, including database connectivity.

**Auth:** None.

```bash
curl https://api.krawl.sh/health
```

**Response:**

```json
{
  "status": "ok",
  "version": "0.2.0",
  "db": "ok",
  "uptime_seconds": 1234.5
}
```

`status` is `ok` when the database is reachable, otherwise `degraded`. `db` is `ok` or `error`.

---

## Research

### `POST /research`

Execute deep research and stream events via SSE. This is the primary endpoint.

**Auth:** API key (or keyless x402 when enabled).

**Request body:**

| Field | Type | Default | Description |
|---|---|---|---|
| `query` | string | *required* | Research query (1 to 2000 chars) |
| `mode` | string | `"deep"` | Research mode. One of: `deep`, `quick`, `crypto`, `token-analysis`, `protocol-research`, `whale-tracking`, `narrative`, `risk-assessment`, `yield-strategy` |
| `depth` | integer | `75` | Research depth intensity (1 to 100) |
| `breadth` | integer | `4` | Parallel queries per depth level (1 to 10) |
| `depth_levels` | integer | `3` | Recursion depth levels (1 to 5) |
| `sources` | string[] | `["exa","x","web"]` | Allowed source types: `exa`, `x`, `github`, `crypto`, `web`, `academic`, `media`, `community` |
| `model` | string \| null | `null` | Override the LLM model for all phases. If `null`, uses server defaults |
| `session_id` | string \| null | `null` | Session ID for conversational follow-up research. Loads prior context when provided |
| `prior_context` | string \| null | `null` | Prior research markdown. Injected by the server (not set directly by callers) |
| `prior_learnings` | object[] \| null | `null` | Prior learnings. Injected by the server |
| `output_schema` | object \| null | `null` | JSON Schema (Draft 7) for structured output extraction |
| `source_policy` | object \| null | `null` | Source filtering rules (see below) |
| `preferred_search_provider` | string \| null | `null` | Web search provider: `exa`, `firecrawl`, `tavily`, `serper`, `brave`, `duckduckgo`, `tinyfish`. Defaults to the server's `WEB_SEARCH_PROVIDER` (`exa` unless configured) |
| `document_context` | string \| null | `null` | Extracted document text. Injected by the server from `/research/upload` |
| `perspectives` | boolean \| null | `null` | STORM perspective-guided research. `null` uses the server default; `true`/`false` forces it per request |

**Source policy:**

```json
{
  "source_policy": {
    "include_domains": ["arxiv.org", "nature.com"],
    "exclude_domains": ["medium.com"],
    "freshness": "30d"
  }
}
```

`freshness` accepts windows like `7d`, `30d`, `90d`, `1y`.

**Response:** SSE stream. See [SSE Events](/docs/sse-events) for all event types.

```bash
curl -N -X POST https://api.krawl.sh/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"query": "AI agents 2025", "mode": "deep", "breadth": 4, "depth_levels": 3}'
```

If `output_schema` is invalid, the request fails with `422` before the stream starts.

### `POST /research/upload`

Research with uploaded document context. Multipart form data. Returns the same SSE stream as `/research`.

**Auth:** API key (or keyless x402 when enabled).

**Form fields:**

| Field | Type | Default | Description |
|---|---|---|---|
| `query` | string | *required* | Research query (1 to 2000 chars) |
| `mode` | string | `"deep"` | Research mode (see modes above) |
| `files` | File[] | `[]` | Up to 5 files, 10MB each. Formats: `.pdf`, `.txt`, `.md`, `.csv`, `.json` |

```bash
curl -N -X POST https://api.krawl.sh/research/upload \
  -H "X-API-Key: your-key" \
  -F "query=Analyze the findings in these documents" \
  -F "mode=deep" \
  -F "files=@report.pdf" \
  -F "files=@data.csv"
```

Empty file slots are ignored. An invalid `mode` or too many files returns `422`.

### `POST /playground`

Run a real research stream with no API key. This is the homepage playground, a marketing and activation surface, not a free API tier.

**Auth:** None. Heavily constrained: forced `quick` mode, short query cap, tiny cost ceiling, per-IP rate limit (`15/hour`, raised from `3/hour` in 2026-06 to fit small teams behind corporate NAT — see `/rate-limits`), and results are NOT persisted. Returns `404` if the playground is disabled.

**Request body:**

| Field | Type | Default | Description |
|---|---|---|---|
| `query` | string | *required* | Query, capped at the server's playground max length |

```bash
curl -N -X POST https://api.krawl.sh/playground \
  -H "Content-Type: application/json" \
  -d '{"query": "What is Uniswap v4?"}'
```

**Response:** SSE stream (same event shape as `/research`).

### `POST /research/compare`

Compare research results to detect changes over time. Returns a structured diff with new findings, removed claims, changed metrics, and a natural language summary.

**Auth:** API key.

**Request body:**

| Field | Type | Default | Description |
|---|---|---|---|
| `query` | string | *required* | Query used to find similar results when `result_ids` is empty (1 to 2000 chars) |
| `result_ids` | string[] | `[]` | Up to 10 result IDs. With 2+, compares those directly. With 1, compares against the most similar prior result. Empty finds the 2 most recent similar results |

```bash
curl -X POST https://api.krawl.sh/research/compare \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"query": "ETH staking yields"}'
```

**Response:**

```json
{
  "old_result_id": "uuid",
  "new_result_id": "uuid",
  "old_query": "ETH staking yields",
  "new_query": "ETH staking yields",
  "old_created_at": "2026-05-01T00:00:00Z",
  "new_created_at": "2026-05-29T00:00:00Z",
  "changes": [],
  "new_findings": [],
  "removed_claims": [],
  "summary": "...",
  "has_changes": true
}
```

### `POST /research/{research_id}/steer`

Inject a steering instruction into an active research session. The `research_id` is the session ID emitted at the start of the SSE stream.

The latest queued instruction applies before the next agent turn in quick mode,
or query-generation call in recursive modes, including child research nodes.
It carries into synthesis/refinement, report scope and requested structured
output without rerunning completed searches. The audit precedes the result so
disconnecting after the report preserves its applied controls.
The `steer` SSE event identifies the instruction consumed at that boundary;
instructions superseded before a boundary do not produce an applied event.
An in-flight provider call continues under its existing request. Durable
pause/resume and branch cancellation are not supported.

**Auth:** API key.

**Request body:**

| Field | Type | Default | Description |
|---|---|---|---|
| `instruction` | string | *required* | Steering instruction (1 to 1000 chars) |

```bash
curl -X POST https://api.krawl.sh/research/550e8400-e29b-41d4-a716-446655440000/steer \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"instruction": "Focus on regulatory risk in the EU"}'
```

**Response:**

```json
{
  "status": "instruction_queued",
  "research_id": "550e8400-e29b-41d4-a716-446655440000",
  "instruction": "Focus on regulatory risk in the EU"
}
```

Returns `404` if there is no active session with that ID.

---

## Async Tasks

Submit research to run server-side without holding an SSE connection, then poll for status.

### `POST /research/async`

Submit a research job to run in the background. Returns immediately with `202`.

**Auth:** API key. Quota is enforced up front, and a per-key cap bounds concurrent fan-out (returns `429` when exceeded).

**Request body:**

| Field | Type | Default | Description |
|---|---|---|---|
| `query` | string | *required* | Research query (1 to 2000 chars) |
| `mode` | string | `"deep"` | Research mode (see modes above) |
| `perspectives` | boolean \| null | `null` | STORM perspective-guided research (`null` uses server default) |
| `webhook_url` | string \| null | `null` | Optional URL to POST a completion notification to (max 2000 chars) |

```bash
curl -X POST https://api.krawl.sh/research/async \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"query": "State of L2 rollups", "mode": "deep"}'
```

**Response (`202`):**

```json
{
  "id": "uuid",
  "query": "State of L2 rollups",
  "mode": "deep",
  "status": "queued",
  "result_id": null,
  "error": null,
  "created_at": "2026-05-29T00:00:00Z",
  "started_at": null,
  "finished_at": null
}
```

`status` is one of `queued`, `running`, `done`, `error`.

### `GET /tasks`

List the caller's async tasks, newest first.

**Auth:** API key.

**Query params:** `limit` (default `50`, clamped to 1 to 100), `offset` (default `0`).

```bash
curl https://api.krawl.sh/tasks?limit=20 \
  -H "X-API-Key: your-key"
```

**Response:** an array of task objects (same shape as above).

### `GET /tasks/{task_id}`

Poll a single async task. When `status` is `done`, fetch the report via `GET /results/{result_id}`.

**Auth:** API key.

```bash
curl https://api.krawl.sh/tasks/uuid \
  -H "X-API-Key: your-key"
```

Returns `404` if the task is not found. Response is a single task object.

---

## OpenAI-compatible

Point any OpenAI SDK (or LangChain, LlamaIndex, etc.) at Krawl by setting `base_url` to this server.

### `POST /v1/chat/completions` and `POST /chat/completions`

OpenAI-compatible Chat Completions. The `model` selects the research mode and the last user message is the research query. Returns a fully cited report as the assistant message. Supports streaming and non-streaming.

**Auth:** API key (or keyless x402 when enabled).

**Request body:**

| Field | Type | Default | Description |
|---|---|---|---|
| `model` | string | `"krawl-deep"` | Maps to a research mode (e.g. `krawl-deep`, `krawl-quick`, `krawl-crypto`) |
| `messages` | object[] | *required* | Chat messages. The last user message becomes the query. Each message has `role`, `content` (string or content-part array), optional `name` |
| `stream` | boolean | `false` | When `true`, returns SSE `chat.completion.chunk` events ending with `[DONE]` |

Unknown fields (`temperature`, `top_p`, `tools`, etc.) are accepted and ignored so existing OpenAI callers work unchanged.

```bash
curl -X POST https://api.krawl.sh/v1/chat/completions \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{
    "model": "krawl-deep",
    "messages": [{"role": "user", "content": "Compare Solana and Aptos"}]
  }'
```

**Response (non-streaming):** a standard OpenAI `chat.completion` object with the report in `choices[0].message.content`. Returns `422` if there is no user message content to research.

### `GET /v1/models` and `GET /models`

List the available `krawl-*` research models in OpenAI's format.

**Auth:** None enforced beyond the default rate limit.

```bash
curl https://api.krawl.sh/v1/models
```

**Response:**

```json
{
  "object": "list",
  "data": [
    {"id": "krawl-deep", "object": "model", "owned_by": "krawl"}
  ]
}
```

---

## Templates

Reusable research prompts with fillable variables.

### `GET /templates`

List all available research templates.

**Auth:** API key.

```bash
curl https://api.krawl.sh/templates \
  -H "X-API-Key: your-key"
```

**Response:** `{"templates": [...]}`

### `GET /templates/{template_id}`

Get a specific template by ID. Returns `404` if not found.

**Auth:** API key.

```bash
curl https://api.krawl.sh/templates/token-deep-dive \
  -H "X-API-Key: your-key"
```

### `POST /templates/{template_id}/research`

Execute a template with filled-in variables. Returns an SSE stream (same as `/research`). The template's own `mode` is used.

**Auth:** API key (or keyless x402 when enabled).

**Request body:**

| Field | Type | Default | Description |
|---|---|---|---|
| `variables` | object | `{}` | Map of variable names to string values used to fill the template |

```bash
curl -N -X POST https://api.krawl.sh/templates/token-deep-dive/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"variables": {"token": "ETH"}}'
```

Returns `404` for an unknown template and `422` if required variables are missing.

---

## Results & Sessions

### `GET /results`

List persisted research results.

**Auth:** API key (results are scoped to the key).

**Query params:** `offset` (default `0`), `limit` (default `50`, 1 to 100).

```bash
curl https://api.krawl.sh/results?limit=10 \
  -H "X-API-Key: your-key"
```

**Response:** `{"results": [...]}`

### `GET /results/{result_id}`

Retrieve a specific result. The `result_id` must be a UUID. Returns `404` if not found.

**Auth:** API key.

```bash
curl https://api.krawl.sh/results/550e8400-e29b-41d4-a716-446655440000 \
  -H "X-API-Key: your-key"
```

### `GET /results/{result_id}/export`

Export a result as HTML, PDF, Markdown, DOCX, or a slide deck.

**Auth:** API key.

**Query params:**

| Param | Type | Default | Description |
|---|---|---|---|
| `format` | string | `html` | One of `pdf`, `html`, `markdown`, `docx`, `slides`, `run-record` |
| `theme` | string | `light` | One of `light`, `dark` (HTML output) |

```bash
curl -L "https://api.krawl.sh/results/550e8400-e29b-41d4-a716-446655440000/export?format=pdf" \
  -H "X-API-Key: your-key" \
  -o report.pdf
```

Returns the rendered file with an appropriate `Content-Type` and `Content-Disposition`. Returns `404` if the result is not found.

### `GET /sessions/{session_id}`

Return all research results in a session, ordered by creation time. The `session_id` must be a UUID.

**Auth:** API key.

```bash
curl https://api.krawl.sh/sessions/550e8400-e29b-41d4-a716-446655440000 \
  -H "X-API-Key: your-key"
```

**Response:**

```json
{
  "session_id": "550e8400-e29b-41d4-a716-446655440000",
  "results": [...]
}
```

Returns `404` if the session does not exist or has no results.

### `GET /audit/{result_id}`

Get the audit trail for a research result: per-step search, extract, learn, gap-detect, synthesize, and verify actions with timing and token usage.

**Auth:** API key.

```bash
curl https://api.krawl.sh/audit/550e8400-e29b-41d4-a716-446655440000 \
  -H "X-API-Key: your-key"
```

**Response:**

```json
{
  "result_id": "550e8400-e29b-41d4-a716-446655440000",
  "entries": [
    {
      "timestamp": "2026-05-29T00:00:00Z",
      "action": "search",
      "tool": "webSearch",
      "query": "...",
      "sources_found": 8,
      "learnings_extracted": 3,
      "model": "...",
      "tokens_used": 1200,
      "duration_ms": 850.0
    }
  ]
}
```

Returns `404` if there is no audit trail for the result.

---

## Search

### `POST /search`

Execute a single-source search and return JSON results. This is not the full research pipeline, it queries one source directly.

**Auth:** API key.

**Request body:**

| Field | Type | Default | Description |
|---|---|---|---|
| `query` | string | *required* | Search query (1 to 2000 chars) |
| `source` | string | `"exa"` | Source to query: `exa`, `x`, `github`, `crypto`, `web`, `academic`, `media` |
| `options` | object | `{}` | Source-specific options (see below) |

Common `options` keys: `num_results` (clamped to 1 to 30, default 8), `category`, `start_date`, `include_domains` (Exa); `from_date`, `to_date`, `include_handles`, `exclude_handles` (X); `token_id` (crypto); `url` (web and media, fetches or transcribes a specific URL).

```bash
curl -X POST https://api.krawl.sh/search \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"query": "restaking protocols", "source": "exa", "options": {"num_results": 10}}'
```

**Response:** shape varies by source. Most sources return `{"sources": [...]}`. X also returns `text`, `citations`, and `tweets`. GitHub adds `repos`. Crypto returns `token`, `tvl`, and/or `search_results`.

```json
{
  "sources": [
    {
      "url": "https://example.com",
      "title": "...",
      "content": "...",
      "published_date": null,
      "source_type": "exa",
      "author": null,
      "favicon": null,
      "credibility_tier": null
    }
  ]
}
```

---

## Lookouts

Scheduled recurring research tasks.

### `POST /lookouts`

Create a new lookout. Enforces a per-key limit (returns `429` when reached).

**Auth:** API key.

**Request body:**

| Field | Type | Default | Description |
|---|---|---|---|
| `query` | string | *required* | Research query (1 to 2000 chars) |
| `mode` | string | `"deep"` | Research mode (see modes above) |
| `schedule` | string | `"daily"` | One of `hourly`, `daily`, `weekly` |
| `webhook_url` | string \| null | `null` | Optional notification URL (max 500 chars, validated against SSRF) |

```bash
curl -X POST https://api.krawl.sh/lookouts \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"query": "Base ecosystem news", "mode": "quick", "schedule": "daily"}'
```

**Response:** the created lookout object.

```json
{
  "id": "uuid",
  "query": "Base ecosystem news",
  "mode": "quick",
  "schedule": "daily",
  "active": true,
  "last_run": null,
  "last_result_id": null,
  "created_at": "2026-05-29T00:00:00Z",
  "webhook_url": null
}
```

### `GET /lookouts`

List lookouts.

**Auth:** API key.

**Query params:** `offset` (default `0`), `limit` (default `50`, 1 to 100).

```bash
curl https://api.krawl.sh/lookouts \
  -H "X-API-Key: your-key"
```

**Response:** `{"lookouts": [...]}`

### `GET /lookouts/{lookout_id}`

Get a specific lookout. UUID required. Returns `404` if not found.

**Auth:** API key.

### `DELETE /lookouts/{lookout_id}`

Delete a lookout. Returns `404` if not found.

**Auth:** API key.

```bash
curl -X DELETE https://api.krawl.sh/lookouts/uuid \
  -H "X-API-Key: your-key"
```

**Response:** `{"deleted": true}`

### `POST /lookouts/{lookout_id}/run`

Manually trigger a lookout run. Rate-limited to `2/minute` and subject to a per-lookout cooldown (returns `429` if triggered too soon after the last run, or `503` if the server is busy).

**Auth:** API key.

```bash
curl -X POST https://api.krawl.sh/lookouts/uuid/run \
  -H "X-API-Key: your-key"
```

**Response:** the run result.

---

## Memory

Cross-session knowledge base of prior research findings.

### `GET /memory`

Search the knowledge base.

**Auth:** API key.

**Query params:** `q` (required, max 500 chars), `limit` (default `20`, 1 to 100).

```bash
curl "https://api.krawl.sh/memory?q=staking&limit=10" \
  -H "X-API-Key: your-key"
```

**Response:**

```json
{
  "query": "staking",
  "count": 3,
  "memories": [
    {
      "id": "uuid",
      "content": "...",
      "source_query": "...",
      "source_urls": ["..."],
      "tags": ["..."],
      "created_at": "2026-05-29T00:00:00Z",
      "result_id": "uuid"
    }
  ]
}
```

Returns `400` if `q` is empty.

### `GET /memory/stats`

Statistics about the knowledge base.

**Auth:** API key.

```bash
curl https://api.krawl.sh/memory/stats \
  -H "X-API-Key: your-key"
```

### `DELETE /memory/{memory_id}`

Delete a specific memory entry. UUID required. Returns `404` if not found.

**Auth:** API key.

```bash
curl -X DELETE https://api.krawl.sh/memory/uuid \
  -H "X-API-Key: your-key"
```

**Response:** `{"deleted": true, "id": "uuid"}`

---

## Ask

### `POST /ask`

NLWeb-compatible natural language query endpoint. Accepts a question and returns a structured answer pointing the caller to the right research endpoint. Optionally streams NLWeb-style SSE events (`start`, `result`, `complete`).

**Auth:** API key.

**Request body:**

| Field | Type | Default | Description |
|---|---|---|---|
| `query` | string | *required* | Natural language question (1 to 2000 chars) |
| `streaming` | boolean | `false` | When `true`, returns SSE NLWeb events instead of JSON |

```bash
curl -X POST https://api.krawl.sh/ask \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"query": "How do I research a token?"}'
```

**Response:**

```json
{
  "_meta": {"response_type": "answer", "version": "0.1", "provider": "krawl.sh"},
  "query": "How do I research a token?",
  "answer": "Use the /research endpoint for deep research on: How do I research a token?",
  "sources": [],
  "suggested_endpoint": "/research",
  "auth": {"method": "api_key", "header": "X-API-Key", "signup": "https://krawl.sh"}
}
```

---

## MCP

### `POST /mcp` and `GET /mcp`

Model Context Protocol server over Streamable HTTP (JSON-RPC 2.0). Agent platforms call `POST /mcp` to `initialize`, `tools/list`, and `tools/call`. Discovery methods are never gated; only `tools/call` execution is authenticated (X-API-Key, or keyless x402 when enabled). `GET /mcp` returns server metadata and the tool list.

See [Agents & MCP](/docs/agents-mcp) for the full integration guide.

```bash
curl -X POST https://api.krawl.sh/mcp \
  -H "Content-Type: application/json" \
  -d '{"jsonrpc": "2.0", "id": 1, "method": "tools/list"}'
```

---

## Auth & Keys

See [Authentication](/docs/authentication) for the complete model. Key issuance and listing are admin-only and require `KRAWL_ADMIN_KEY`.

### `POST /auth/demo-key`

Generate a rate-limited demo API key for testing. No auth required. The key is hashed at rest and expires in 24 hours. Rate-limited to `3/hour`.

```bash
curl -X POST https://api.krawl.sh/auth/demo-key
```

**Response:**

```json
{
  "api_key": "krawl_demo_...",
  "type": "demo",
  "plan": "demo",
  "rate_limit": "...",
  "monthly_search_credits": 0,
  "monthly_deep_krawls": 0,
  "expires_in": "24 hours",
  "note": "Demo key for testing. Sign up at krawl.sh for full access."
}
```

### `POST /auth/keys`

Issue a new API key. The raw key is returned exactly once.

**Auth:** Admin (`KRAWL_ADMIN_KEY`).

**Request body:**

| Field | Type | Default | Description |
|---|---|---|---|
| `plan` | string | `"free"` | Plan name: `free`, `payg`, `pro`, `test`. `demo` is not issuable here |
| `name` | string | `""` | Human-readable label (max 100 chars) |
| `owner_email` | string \| null | `null` | Owner email (validated format, max 255 chars) |

```bash
curl -X POST https://api.krawl.sh/auth/keys \
  -H "Content-Type: application/json" \
  -H "X-API-Key: $KRAWL_ADMIN_KEY" \
  -d '{"plan": "pro", "name": "acme prod", "owner_email": "ops@acme.com"}'
```

### `GET /auth/keys`

List API key metadata. Never returns raw keys.

**Auth:** Admin (`KRAWL_ADMIN_KEY`).

**Query params:** `owner_email` (optional, max 255 chars).

```bash
curl "https://api.krawl.sh/auth/keys?owner_email=ops@acme.com" \
  -H "X-API-Key: $KRAWL_ADMIN_KEY"
```

**Response:** `{"keys": [{"prefix": "...", "plan": "...", "name": "...", "owner_email": "...", "active": true, "created_at": "...", "last_used_at": "...", "expires_at": "...", "revoked_at": "..."}]}`

### `DELETE /auth/keys`

Revoke the `krawl_` key presented in the `X-API-Key` header (self-service). Rate-limited to `20/minute`.

**Auth:** the key being revoked.

```bash
curl -X DELETE https://api.krawl.sh/auth/keys \
  -H "X-API-Key: krawl_live_..."
```

**Response:** `{"revoked": true}`. Returns `404` if the key is not found or already revoked.

---

## Usage & Billing

### `GET /usage`

Return the caller's usage for the current (or given) month plus plan quotas. Requires a `krawl_` API key. Rate-limited to `60/minute`.

**Auth:** `krawl_` API key.

**Query params:** `period` (optional, format `YYYY-MM`).

```bash
curl https://api.krawl.sh/usage \
  -H "X-API-Key: krawl_live_..."
```

**Response:**

```json
{
  "plan": "pro",
  "period": "2026-05",
  "usage": {"deep_krawls": 12, "search_credits": 340, "total_tokens": 1200000, "cost_usd": 4.231},
  "limits": {"monthly_deep_krawls": 100, "monthly_search_credits": 5000, "rate_limit": "..."},
  "remaining": {"deep_krawls": 88, "search_credits": 4660}
}
```

`remaining` values are `null` for unlimited (negative-limit) plans.

### `GET /usage/events`

Return recent per-request usage events for the caller's key. Rate-limited to `60/minute`.

**Auth:** `krawl_` API key.

**Query params:** `limit` (default `50`, 1 to 200).

```bash
curl https://api.krawl.sh/usage/events \
  -H "X-API-Key: krawl_live_..."
```

**Response:** `{"events": [...]}`

### `POST /billing/checkout`

Start a Stripe Checkout session to upgrade the caller's key to Pro. Only `krawl_live_` keys can be upgraded. Rate-limited to `10/minute`. Returns `503` if billing is not enabled.

**Auth:** `krawl_live_` API key.

**Request body:**

| Field | Type | Default | Description |
|---|---|---|---|
| `email` | string \| null | `null` | Customer email (max 255 chars). Falls back to the key's owner email |

```bash
curl -X POST https://api.krawl.sh/billing/checkout \
  -H "Content-Type: application/json" \
  -H "X-API-Key: krawl_live_..." \
  -d '{"email": "ops@acme.com"}'
```

**Response:** a Stripe Checkout session (includes the hosted checkout URL).

### `POST /billing/portal`

Open the Stripe billing portal for the caller's linked customer. Rate-limited to `10/minute`. Returns `400` if no billing account is linked, `503` if billing is disabled.

**Auth:** `krawl_` API key.

```bash
curl -X POST https://api.krawl.sh/billing/portal \
  -H "X-API-Key: krawl_live_..."
```

**Response:** a Stripe billing portal session (includes the portal URL).

### `POST /billing/webhook`

Consume signature-verified Stripe webhook events and apply plan changes. Called by Stripe, not by API clients. Requires `STRIPE_WEBHOOK_SECRET` and a valid `stripe-signature` header.

**Auth:** Stripe signature.

**Response:** `{"received": true, ...}`. Returns `400` on invalid payload or signature, `503` for retryable reconciliation errors.

---

## Discovery

OAuth and x402 metadata that let agents discover how to authenticate and pay.

### `GET /.well-known/oauth-protected-resource`

OAuth 2.0 Protected Resource metadata (RFC 9728).

**Auth:** None.

```bash
curl https://api.krawl.sh/.well-known/oauth-protected-resource
```

**Response:**

```json
{
  "resource": "https://api.krawl.sh",
  "authorization_servers": ["https://api.krawl.sh"],
  "scopes_supported": ["research:read", "research:write", "results:read"],
  "bearer_methods_supported": ["header"],
  "resource_documentation": "https://docs.krawl.sh"
}
```

### `GET /.well-known/oauth-authorization-server`

OAuth 2.0 Authorization Server metadata (RFC 8414) plus a WorkOS `agent_auth` discovery block.

**Auth:** None.

```bash
curl https://api.krawl.sh/.well-known/oauth-authorization-server
```

### `GET /discovery/resources`

Agent-payable resource discovery for x402 / MPP. Lists the API resources an agent can pay for per-request, with price, network, and asset, so an autonomous agent can discover costs without first triggering a `402`.

**Auth:** None.

```bash
curl https://api.krawl.sh/discovery/resources
```

**Response:**

```json
{
  "x402Version": 1,
  "enabled": true,
  "facilitator": "...",
  "resources": [
    {
      "resource": "https://api.krawl.sh/research",
      "method": "POST",
      "description": "Deep multi-source research report (cited markdown).",
      "x402": {
        "scheme": "exact",
        "network": "...",
        "asset": "...",
        "maxAmountRequired": "...",
        "priceUsd": 0.0,
        "payTo": null
      }
    }
  ],
  "payment_methods": ["x402", "api_key"],
  "auth_docs": "https://krawl.sh/auth.md"
}
```

---

## Internal / experimental

Agentic commerce endpoints (`/checkout-sessions`, `/checkout_sessions`, `/checkout_sessions/{session_id}`, `/agentic_commerce/delegate_payment`) implement UCP/ACP checkout flows. They are internal and excluded from the public OpenAPI schema, and are subject to change.

## Evidence time and access

Research sources carry optional `retrieved_at` (UTC), `published_date`,
`updated_date`, date provenance, `access_status`, `source_record_id` and
`temporal_status`. Missing fields on older reports mean unknown; never assign
export time as retrieval time. Access states are `inspected_text`,
`partial_excerpt`, `provider_summary`, `metadata_only`, `fetch_failed`, or
`unknown`. These describe available evidence, not claim support or truth.
A recent fetch does not make an old article current. Summary/metadata-only
results do not establish that the original text was read.

Optional `source_policy.as_of` sets a UTC evidence cutoff; dates after it are
quarantined in `quarantined_sources`, outside synthesis. Without it the cutoff
is research start. This is not a historical web archive; undated content
remains uncertain. See [evidence methodology](https://docs.krawl.sh/docs/trust#evidence-time-and-access).

## Evidence coverage

The optional `evidence_coverage` result field gives descriptive counts of unique
exact URLs discovered, explicitly fetched, retained, referenced in outgoing
model input, and cited. Stages overlap: a retained search snippet may never
have had its original page fetched. It also reports explicit failed/unavailable
fetches, unknown fetch/access states, retained-host concentration, missing
publication dates and their known range. A model-input reference may contain
only metadata or a short excerpt; a citation is not a support verdict.

These counts do not establish representative sampling or correct selection
bias. Hostnames do not prove independent origins; mirrors may share a publisher.
Old reports without this field have unknown coverage. No extra LLM call is made;
the measured instrumentation overhead is reported as `measurement_overhead_ms`,
not a guarantee of zero added latency. See [evidence methodology](https://docs.krawl.sh/docs/trust#evidence-coverage).

---


# SSE Events

Source: https://docs.krawl.sh/docs/sse-events

The `/research` and `/research/upload` endpoints return Server-Sent Events (SSE). Each event has a `type` and `data` field.

## Connection

```bash
curl -N -X POST https://api.krawl.sh/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"query": "AI agents", "mode": "deep"}'
```

Events arrive as:
```
event: session
data: {"session_id": "uuid"}

event: plan
data: {"status": {...}, "plan": [...]}

event: source
data: {"url": "...", "title": "...", "tool": "webSearch"}

event: report_chunk
data: {"text": "chunk of markdown..."}

event: result
data: {"markdown": "full report...", "sources": [...]}

event: done
data: {}
```

Keepalive pings are sent every 10 seconds.

Research streams include a sequential `id` on each event. Reconnect to
`GET /research/{session_id}/events` with the last received `Last-Event-ID` to
replay later buffered events. Replaying events does not restart research or
repeat provider calls; durable execution recovery is separate.

## Catalog at a glance

Every event Krawl emits, when it fires, and which renderer on the web app
consumes it. The renderer column references components under
`apps/web/src/components/research/` — use it to map SSE → UI when wiring a
new client.

| Event | When it fires | Primary renderer | Consumed by |
|---|---|---|---|
| `session` | First frame on stream open | — | persisted in `useSession()`; powers steer + follow-ups |
| `memory_recall` | Before planning, if prior memories match | `<MemoryRecallStrip />` | shows the recalled-from-KB pills |
| `plan` | After planning phase (Sonnet 4.6) | `<PlanCard />` | top-of-stream plan topic list |
| `perspectives` | After STORM perspective discovery (Haiku 4.5) | `<PerspectivesBoard />` | 3–5 perspective chips |
| `node_started` | Each new node in the recursion tree | `<ResearchTree />` | adds a node to the tree visualisation |
| `query` | Each search query that goes out | `<EventFeed />` | one row per query in the live feed |
| `thinking` | Reflection between tool calls (flat/quick mode) | `<ThinkingBubble />` | renders the inline thought |
| `source` | Each new source discovered | `<SourceList />` | adds the URL to the cited-sources list |
| `content` | Content extracted from a source (flat mode) | `<SourceList />` | source row gains a `content_length` chip |
| `x_search` | X/Twitter aggregated search frame | `<XSearchCard />` | X-only card with tweet count + summary |
| `browse_page` | Page fetched via `browsePage` tool | `<SourceList />` | source row marked `(browsed)` |
| `learning` | New extracted learning passes dedup | `<LearningsList />` | learning ticker, sorted by confidence |
| `gap_detected` | Sonnet detects a gap; will trigger recursion | `<GapsList />` | gap chip in the right rail |
| `outline_updated` | Dynamic outline refreshed with new evidence | `<OutlinePane />` | report TOC live-updates |
| `node_completed` | Node finishes (own sources + learnings counted) | `<ResearchTree />` | node turns green, shows counts |
| `report_chunk` | Streaming markdown token frame (many per run) | `<ReportViewer />` | typewriter into the main report pane |
| `citation_verified` | A `[n]` cite is verified against the fetched text | `<ReportViewer />` | renders verified badge on the inline citation |
| `chart` | Time-series data extracted (crypto modes mainly) | `<ChartRenderer />` | inline chart in the report |
| `structured_output` | Output extracted per `output_schema` | `<StructuredOutputPanel />` | structured fields panel with citations |
| `review_pass` | Multi-pass review scored the draft | `<ReviewBanner />` | review pass + score banner |
| `audit` | Full audit trail flushed before `result` | `<AuditTrail />` | the `/audit/{id}` viewer |
| `steer` | A steer instruction was accepted | `<SteerPill />` | toast + steer history entry |
| `document` | An uploaded doc finished processing | `<UploadedDocs />` | upload card shows `chars_extracted` |
| `usage` | Terminal usage + cost telemetry | `<UsageFooter />` | totals + per-model breakdown footer |
| `changes_since_last` | Diff vs previous result in the session | `<ChangesPanel />` | "what changed" summary card |
| `result` | Final result frame — markdown + sources + learnings | `<ReportViewer />` | final report state; saved actions wait for the terminal usage acknowledgement |
| `done` | Stream complete | `<EventFeed />` | flips status to `done` |
| `error` | Stream-fatal error | `<EventFeed />` | error toast + retry CTA |

## Event Types

### `session`

Emitted first. Contains the session ID for steering and follow-up research.

```json
{"session_id": "550e8400-e29b-41d4-a716-446655440000"}
```

### `plan`

The research plan generated by the planning phase.

```json
{
  "status": {"title": "Research plan ready, starting recursive deep research"},
  "plan": [
    {
      "title": "AI Agent Architectures",
      "todos": [
        "Research current AI agent frameworks",
        "Find recent papers on agent architectures"
      ]
    }
  ]
}
```

### `perspectives`

STORM perspective discovery results. 3–5 perspectives for multi-angle research.

```json
{
  "perspectives": [
    {
      "name": "AI Researcher",
      "description": "Focuses on technical architecture and capability benchmarks",
      "key_questions": ["What architectures dominate?", "How do agents handle tool use?"]
    }
  ]
}
```

### `thinking`

Agent's internal reasoning between tool calls (flat/quick mode).

```json
{
  "thought": "The search results cover frameworks but not benchmarks. I should search for agent evaluation methods.",
  "nextStep": "Searching for AI agent benchmarks"
}
```

### `query`

A search query being executed.

```json
{"query": "AI agent frameworks 2025", "tool": "webSearch"}
```

### `source`

A new source discovered during search.

```json
{
  "url": "https://arxiv.org/abs/2401.00001",
  "title": "A Survey of AI Agents",
  "tool": "webSearch"
}
```

### `content`

Content extracted from a source (flat mode).

```json
{"url": "https://...", "content": "Extracted text...", "chars": 3000}
```

### `x_search`

X/Twitter search results.

```json
{
  "query": "AI agents",
  "text": "Summary of X discourse...",
  "tweets_count": 12,
  "citations": ["https://x.com/..."]
}
```

### `browse_page`

Page content fetched via browsePage tool.

```json
{"url": "https://...", "title": "...", "content_length": 5000}
```

### `report_chunk`

Streaming markdown chunk during synthesis. Arrives many times.

```json
{"text": "## Introduction\n\nAI agents have evolved..."}
```

### `result`

The final research result. Contains the complete report and all sources.

It includes a logical `id`, `query`, `mode`, `created_at`, measured
`duration_seconds`, and `persisted: false`. This identity supports inline
evidence-pack export, including ZDR runs. It does not imply that a saved report
exists. Only enable saved-report downloads and sharing after the terminal
`usage.result_id` acknowledges successful persistence.

```json
{
  "markdown": "# Full Research Report\n\n...",
  "sources": [
    {"url": "...", "title": "...", "source_type": "exa", "content": "..."}
  ],
  "total_steps": 24,
  "total_learnings": 18,
  "model": "bedrock/us.anthropic.claude-opus-4-7",
  "session_id": "...",
  "learnings": [...],
  "structured": null
}
```

### `error`

An error occurred during research.

```json
{"message": "Research failed unexpectedly"}
```

### `done`

Research stream is complete. No more events will be sent.

```json
{}
```

### `node_started`

A new node in the recursive research tree has started.

```json
{
  "node_id": "...",
  "query": "AI agent frameworks",
  "depth": 3,
  "breadth": 4,
  "parent_id": null
}
```

### `node_completed`

A research tree node has completed.

```json
{
  "node_id": "...",
  "learnings_count": 5,
  "sources_found": 8,
  "gaps_found": 2
}
```

### `learning`

A new learning extracted from search results.

```json
{
  "content": "GPT-4 based agents can solve 78% of SWE-bench tasks",
  "source_url": "https://...",
  "confidence": 0.95
}
```

### `gap_detected`

A knowledge gap identified during research.

```json
{
  "question": "How do agents handle long-horizon planning?",
  "priority": 0.9,
  "suggested_tools": ["webSearch", "academicSearch"]
}
```

### `outline_updated`

The dynamic report outline has been updated with new evidence.

```json
{
  "sections": [
    {"title": "Agent Architectures", "evidence_count": 5},
    {"title": "Tool Use Patterns", "evidence_count": 3}
  ]
}
```

### `citation_verified`

A citation has been verified against the fetched source content.

```json
{
  "claim": "GPT-4 achieves 78% on SWE-bench",
  "source_url": "https://...",
  "verified": true,
  "source_excerpt": "Our evaluation shows GPT-4..."
}
```

### `audit`

The full audit trail for the research session. When present, it precedes
`result` so a disconnect immediately after the report preserves applied
controls in the saved audit.

```json
{
  "entries": [
    {
      "timestamp": "2025-01-15T10:30:00Z",
      "action": "search",
      "tool": "webSearch",
      "sources_found": 8
    }
  ]
}
```

### `steer`

The latest queued steering instruction was consumed at a quick agent turn,
recursive query or synthesis/refinement boundary. It applies to that call and subsequent child
queries/report scope; it cannot alter a provider request already in flight.
Instructions superseded before the boundary are not emitted as applied events.
The consumed directive also scopes structured output when `output_schema` is
requested; the original research query remains available as context.

```json
{"instruction": "Focus more on security aspects"}
```

### `chart`

Data visualization extracted from research findings.

```json
{
  "chart_type": "bar",
  "title": "Market Cap Comparison",
  "data": {...}
}
```

### `structured_output`

Structured data extracted per the provided `output_schema`.

```json
{
  "fields": {
    "market_cap": {
      "value": "$400B",
      "confidence": 0.95,
      "citation": "https://...",
      "reasoning": "Multiple sources confirm..."
    }
  }
}
```

### `review_pass`

Multi-pass synthesis review results.

```json
{
  "pass_number": 1,
  "score": 0.72,
  "issues_count": 3
}
```

### `memory_recall`

Prior knowledge recalled from the cross-session knowledge base.

```json
{
  "memories_count": 5,
  "memories": [
    {"content": "Ethereum merged to PoS in Sep 2022", "tags": ["ethereum"]}
  ]
}
```

### `document`

Document upload processing status.

```json
{
  "filename": "report.pdf",
  "chars_extracted": 15000,
  "status": "processed"
}
```

### `usage`

Terminal usage and cost telemetry, emitted near the end of a run. The meter snapshot is annotated with the pricing tier the request was billed at.

A retained run adds `result_id` only after saving its report successfully
(`null` on a failed save). ZDR and playground runs do not acknowledge a saved
result. Incremental snapshots with `partial: true` do not confirm persistence.

```json
{
  "prompt_tokens": 48213,
  "completion_tokens": 9120,
  "total_tokens": 57333,
  "cost_usd": 0.412,
  "llm_calls": 24,
  "search_calls": 11,
  "source_count": 57,
  "cost_ceiling_usd": 1.5,
  "ceiling_hit": false,
  "by_model": {
    "bedrock/us.anthropic.claude-opus-4-7": {"calls": 1, "total_tokens": 18400, "cost_usd": 0.31}
  },
  "pricing": {
    "tier": "deep",
    "billed_as": "deep_krawl",
    "cost_ceiling_usd": 2.5,
    "x402_price_usd": 5.0
  }
}
```

### `changes_since_last`

Auto-comparison with the previous result in the same session (emitted for follow-up research).

```json
{
  "previous_result_id": "...",
  "has_changes": true,
  "new_findings": ["New finding 1"],
  "removed_claims": [],
  "changes": [{"metric": "price", "old_value": "$50k", "new_value": "$55k"}],
  "summary": "Since your last research: price increased..."
}
```

## Event Order (Deep Mode)

Typical event sequence for a deep research session:

1. `session` — session ID
2. `memory_recall` — prior knowledge (if any)
3. `plan` — research plan
4. `perspectives` — STORM perspectives (if query is complex enough)
5. `node_started` → `query` → `source` (repeated) → `learning` → `node_completed` (per tree node)
6. `gap_detected` → recurse (depth-1)
7. `outline_updated`
8. `report_chunk` (many) — streaming synthesis
9. `review_pass` — quality review
10. `chart` — data visualizations (if applicable)
11. `structured_output` — structured data (if output_schema provided)
12. `citation_verified` (multiple)
13. `audit` — full audit trail
14. `result` — final result
15. `changes_since_last` — comparison (if follow-up session)
16. `done` — stream complete

## JavaScript Client Example

```javascript
const evtSource = new EventSource('/research', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'X-API-Key': 'your-key',
  },
  body: JSON.stringify({ query: 'AI agents', mode: 'deep' }),
});

// Note: EventSource only supports GET. For POST, use fetch with ReadableStream:
const response = await fetch('https://api.krawl.sh/research', {
  method: 'POST',
  headers: {
    'Content-Type': 'application/json',
    'X-API-Key': 'your-key',
  },
  body: JSON.stringify({ query: 'AI agents', mode: 'deep' }),
});

const reader = response.body.getReader();
const decoder = new TextDecoder();
let buffer = '';

while (true) {
  const { done, value } = await reader.read();
  if (done) break;

  buffer += decoder.decode(value, { stream: true });
  const lines = buffer.split('\n');
  buffer = lines.pop() || '';

  for (const line of lines) {
    if (line.startsWith('event: ')) {
      const eventType = line.slice(7);
      // Next line is data
    }
    if (line.startsWith('data: ')) {
      const data = JSON.parse(line.slice(6));
      console.log(eventType, data);
    }
  }
}
```

## Evidence time and access

Research sources carry optional `retrieved_at` (UTC), `published_date`,
`updated_date`, date provenance, `access_status`, `source_record_id` and
`temporal_status`. Missing fields on older reports mean unknown; never assign
export time as retrieval time. Access states are `inspected_text`,
`partial_excerpt`, `provider_summary`, `metadata_only`, `fetch_failed`, or
`unknown`. These describe available evidence, not claim support or truth.
A recent fetch does not make an old article current. Summary/metadata-only
results do not establish that the original text was read.

Optional `source_policy.as_of` sets a UTC evidence cutoff; dates after it are
quarantined in `quarantined_sources`, outside synthesis. Without it the cutoff
is research start. This is not a historical web archive; undated content
remains uncertain. See [evidence methodology](https://docs.krawl.sh/docs/trust#evidence-time-and-access).

## Evidence coverage

The optional `evidence_coverage` result field gives descriptive counts of unique
exact URLs discovered, explicitly fetched, retained, referenced in outgoing
model input, and cited. Stages overlap: a retained search snippet may never
have had its original page fetched. It also reports explicit failed/unavailable
fetches, unknown fetch/access states, retained-host concentration, missing
publication dates and their known range. A model-input reference may contain
only metadata or a short excerpt; a citation is not a support verdict.

These counts do not establish representative sampling or correct selection
bias. Hostnames do not prove independent origins; mirrors may share a publisher.
Old reports without this field have unknown coverage. No extra LLM call is made;
the measured instrumentation overhead is reported as `measurement_overhead_ms`,
not a guarantee of zero added latency. See [evidence methodology](https://docs.krawl.sh/docs/trust#evidence-coverage).

---


# Benchmarks

Source: https://docs.krawl.sh/docs/benchmarks

This page is the audit trail for any "Krawl beats X" claim. Every score
below is dated, has the run config it was generated from, and links to
the scoring code on GitHub. If a number on a marketing page disagrees
with a number here, this page wins.

> **Disclosure.** We publish the methodology and the harness, but we are
> not a neutral third party. Re-run the harness yourself to verify; the
> code is MIT and the eval data is public. See
> [Trust](/docs/trust) for failure modes and refund policy.

## Public benchmarks

### FRAMES (multi-hop QA)

[FRAMES](https://huggingface.co/datasets/google/frames-benchmark) is
Google's multi-hop QA bench (824 questions, multi-doc reasoning). The
benchmark is the right hardness ceiling for an open-web research agent.

| Configuration | Score (0–1) | Date | Harness |
|---|---|---|---|
| `krawl-deep` (Opus 4.7 synthesis, breadth 4, depth 3) | _publishing_ | _eval pending_ | [`scripts/bench/frames.py`](https://github.com/ravidsrk/krawl/blob/main/scripts/bench/frames.py) |
| `krawl-quick` (Sonnet 4.6 flat loop) | _publishing_ | _eval pending_ | same harness, `--mode quick` |

Scores publish on the next eval run; the harness is checked in. Run it
yourself:

```bash
git clone https://github.com/ravidsrk/krawl && cd krawl
export KRAWL_API_KEY=krawl_demo_xxx
python scripts/bench/frames.py --mode deep --limit 824
```

### SimpleQA (factual QA)

[SimpleQA](https://openai.com/index/introducing-simpleqa) is OpenAI's
adversarial single-hop bench (4,326 questions). Used as a freshness +
factuality canary — short questions, hard sources.

| Configuration | Score (0–1) | Date | Harness |
|---|---|---|---|
| `krawl-quick` | _publishing_ | _eval pending_ | [`scripts/bench/simpleqa.py`](https://github.com/ravidsrk/krawl/blob/main/scripts/bench/simpleqa.py) |
| `krawl-deep` | _publishing_ | _eval pending_ | same harness, `--mode deep` |

### FinSearchComp (financial research)

[FinSearchComp](https://arxiv.org/abs/2406.14260) tests financial
research over heterogeneous sources (filings, news, prices). The right
bench for the finance / crypto mode mix.

| Configuration | Score (0–1) | Date | Harness |
|---|---|---|---|
| `krawl-crypto` (`crypto` mode) | _publishing_ | _eval pending_ | [`scripts/bench/finsearchcomp.py`](https://github.com/ravidsrk/krawl/blob/main/scripts/bench/finsearchcomp.py) |

The numbers will land here once the eval run completes; we do not
prepublish "expected" scores.

## Krawl Crypto Research Bench

Public benchmarks are general-purpose. Crypto research has its own
failure modes — claims that flip with on-chain state, sources that
disagree on supply numbers, narratives that move faster than the eval
loop. The Krawl Crypto Research Bench (KCRB) is our published bench for
that surface.

### Why a new bench

FRAMES and FinSearchComp don't cover any of:

- **On-chain freshness** — TVL, holder counts, active addresses that
  change daily.
- **Multi-source disagreement** — CoinGecko vs DeFi Llama vs the
  protocol's own dashboards.
- **Narrative tracking** — claims that depend on the X/farcaster
  discourse around an event.

KCRB tests for those three explicitly.

### Methodology

- **Questions**: 400 dated questions, 100 per category — token
  fundamentals, protocol mechanics, on-chain activity, narrative
  tracking.
- **Sources**: each question carries a `verified_answer` and a
  `verification_url` set captured on the question's `as_of` date. Both
  are public.
- **Scoring**: 0–1 per question, breakdown:
  - `factual` — does the answer match the verified answer? (0/1)
  - `cited` — does every claim have an `[n]` that resolves to a fetched
    source? (0/1)
  - `freshness` — was at least one source dated within the question's
    freshness window? (0/1)
  - `disagreement` — for multi-source questions, did the report surface
    the disagreement explicitly? (0/1)
- **Aggregate**: weighted mean (0.5 factual, 0.2 cited, 0.2 freshness,
  0.1 disagreement). The weights are fixed in the harness and printed in
  the result header.

### Harness + scoring code

The bench is open-source MIT. Code, eval data, and the scoring rubric
all live in-repo:

- Harness: [`scripts/bench/kcrb.py`](https://github.com/ravidsrk/krawl/blob/main/scripts/bench/kcrb.py)
- Eval data: [`scripts/bench/kcrb_questions.jsonl`](https://github.com/ravidsrk/krawl/blob/main/scripts/bench/kcrb_questions.jsonl)
- Scoring: [`scripts/bench/kcrb_score.py`](https://github.com/ravidsrk/krawl/blob/main/scripts/bench/kcrb_score.py)
- Leaderboard JSON: [`scripts/bench/kcrb_leaderboard.json`](https://github.com/ravidsrk/krawl/blob/main/scripts/bench/kcrb_leaderboard.json)

Run it yourself:

```bash
export KRAWL_API_KEY=krawl_demo_xxx
python scripts/bench/kcrb.py --mode crypto --limit 400
python scripts/bench/kcrb_score.py --runs runs/*.jsonl
```

### Leaderboard (KCRB v0.1)

The leaderboard is reproducible from the artifacts above. Submit a run
by opening a PR with a `runs/<system>-<date>.jsonl` file from the
harness.

| System | Date | Factual | Cited | Freshness | Disagreement | Aggregate |
|---|---|---|---|---|---|---|
| Krawl (`crypto` mode) | _publishing_ | _eval pending_ | _eval pending_ | _eval pending_ | _eval pending_ | _eval pending_ |
| Manus | _open invite_ | — | — | — | — | — |
| Exa research API | _open invite_ | — | — | — | — | — |
| Perplexity (deep research) | _open invite_ | — | — | — | — | — |

Where a competitor doesn't expose a deep-research API, we run their
chat surface against the same questions; that footnote sticks with the
score in the leaderboard JSON.

## Methodology notes

- **No mocked sources.** All harnesses hit the real API surface (Krawl
  staging or the competitor's prod), no fakes, no replayed traces.
- **Reproducible seeds.** Where the engine exposes a seed, the harness
  sets it. Where it doesn't, the harness re-runs each question N=3
  times and takes the modal verified answer.
- **Dated.** Every published score has the `as_of` date for the eval
  data and the `run_at` date for the system's response. Both live in
  the JSON.
- **Cost reporting.** The harness logs the total `usage` event for each
  run; the leaderboard JSON includes per-run cost so "Krawl beat X but
  spent 10×" is provable.

If a number on this page contradicts code on `main`, the code wins.
Open a PR.

---


# Trust

Source: https://docs.krawl.sh/docs/trust

Krawl is a deep-research API. The result is a cited markdown report. This
page describes how we get there, where the failure modes live, and what we
do when something goes wrong.

If a claim below contradicts what the engine actually does, treat the code
as the source of truth and open a docs issue.

## Methodology

### What "deep research" means here

A deep run executes a recursive breadth × depth tree search:

1. **Plan** (Sonnet 4.6). A structured research plan with topics and tasks.
2. **STORM perspectives** (Haiku 4.5). 3–5 perspectives with key questions
   so the search tree explores multiple angles. Skipped for very short
   queries.
3. **Recursive tree** (Haiku 4.5 for queries; Sonnet 4.6 for extraction
   and gap analysis). Generate N queries per depth level, execute in
   parallel across the enabled tools, extract structured learnings,
   deduplicate (Jaccard ≥ 0.7), detect gaps, recurse with
   `depth - 1, breadth / 2`.
4. **Verified synthesis** (Opus 4.7, exactly one synthesis call). Built
   on a retrieve-then-cite block — source metadata distinguishes original text, excerpts, provider summaries and
   links without inspected content. A collected URL alone does not verify a claim.
5. **Gap check + review/revise** (Sonnet 4.6 / Opus 4.7). The draft is
   scored 0–1 by an independent review pass; if &lt; 0.8, one or two
   revision passes run. Hard cap: two revisions.

Full pipeline detail: [Architecture](/docs/architecture).

### What "cited" means

Every numeric `[n]` in the report is a citation pointer into the source
list. The synthesis prompt enforces two rules that the engine then verifies:

- **Source-bounded**: the model is given a `<sources>` block and may cite
  only those sources. A `[n]` outside the block is rejected and the chunk
  is re-asked.
- **Verified against text**: each `[n]` is verified against the fetched
  source content. The verification stream emits a `citation_verified`
  event (`verified: true|false`) per citation; unverified citations
  trigger a re-pass.

A real source counter shows up in the `result` event under
`sources_used`; if you see fewer sources used than were fetched, that's
normal — only sources the model actually cited are surfaced.

### What we don't claim

- We do not claim every fact in the report is true. We claim each cited
  fact came from a source we retrieved on the call and was verified
  against that source's text. The source itself can still be wrong.
- We do not pin a freshness guarantee on a deep run. The engine uses live
  search providers (Exa, Firecrawl, X, GitHub, CoinGecko, DeFi Llama,
  Semantic Scholar, PubMed, Reddit, HN, Wikipedia, Tavily, Serper, Brave,
  DuckDuckGo), but provider freshness varies. Where it matters,
  cross-check against the source link.
- We do not guarantee determinism. Same query twice can land on
  different sources, and Opus's synthesis is sampled. If you need
  reproducibility, set `seed` (where supported) and pin the source list
  via `output_schema`.

## Failure modes (named)

Knowing what can go wrong is half of trust. Krawl emits these as
structured failure types — agents can branch on them.

| Failure | Where it surfaces | What it means | What to do |
|---|---|---|---|
| `provider_4xx` | per-tool, in audit | One search provider rate-limited or refused | Engine retries on a different provider; report often still ships |
| `provider_5xx` | per-tool, in audit | Provider outage | Engine fails over (Bedrock → Anthropic → cross-provider) |
| `circuit_open` | `error` event | Bedrock circuit breaker tripped (3 fails) | Anthropic direct fallback engages for 5 min |
| `cost_ceiling_hit` | `usage` event (`ceiling_hit: true`) | Per-request cost ceiling reached mid-synthesis | Report finalises with whatever's gathered so far |
| `gap_unresolved` | `gap_detected` w/o subsequent `node_completed` | A gap couldn't be filled before recursion exhausted | Report ships marked with `gaps_unresolved > 0` |
| `citation_unverified` | `citation_verified: false` | A `[n]` failed verification against source text | Engine re-asks the surrounding chunk; if persistent, citation is stripped |
| `review_below_threshold` | `review_pass` `score < 0.8` after pass 2 | Two revisions still didn't clear the bar | Report ships with the best draft + a `quality_flag` |
| `429_rate_limited` | top-level 429 | Caller hit the per-IP / per-key rate limit | Honor `Retry-After` and back off (see [Rate Limits](/docs/rate-limits)) |
| `402_payment_required` | top-level 402 | x402 challenge | Sign + retry with `X-PAYMENT` (see [x402](/docs/payments-x402)) |

The full audit trail of any specific run is at
[`GET /audit/{result_id}`](/docs/api-reference#get-audit-result_id).

## Refund policy

x402 payments are **pay-per-use and non-refundable** once settlement
succeeds. This is stated on the 402 challenge itself
(`description: "… (pay-per-use, non-refundable)"`) so an agent can decide
before signing.

Two specific exceptions:

- **Pre-settlement 503.** If the server is at capacity, the 402 response
  carries an empty `accepts` array and is rejected **before** the
  facilitator's `/settle` runs. No funds move.
- **Post-settlement engine failure.** If a paid request settles and then
  the engine fails with a 5xx (not a 4xx your client caused), we'll
  credit the request manually on review — open a ticket with the
  transaction hash from `X-PAYMENT-RESPONSE`. We don't promise SLA-grade
  automation here; it's a goodwill credit.

API-key requests follow the plan's terms — see
[Authentication](/docs/authentication) for plan-scoped quotas and
overage behavior.

## Audit and verification surfaces

Two product surfaces let an agent or operator verify a specific run:

- **Per-result audit log**:
  [`GET /audit/{result_id}`](/docs/api-reference#get-audit-result_id)
  returns every search, browse, extraction, and synthesis step for the
  run — what got called, with what argument, what came back.
- **Citation verification events**: the SSE stream emits
  `citation_verified` for every `[n]` in the report, with the source URL
  and the supporting excerpt. See
  [SSE Events → `citation_verified`](/docs/sse-events#citation_verified).

For benchmark transparency — public scores on third-party benches and the
methodology behind the Krawl Crypto Research Bench — see
[Benchmarks](/docs/benchmarks).

## Security and data handling

- **Auth**: `X-API-Key` is HMAC-compared. Demo keys expire and are
  metered separately.
- **Headers**: HSTS (2y, includeSubDomains), `X-Frame-Options: DENY`,
  `X-Content-Type-Options: nosniff`.
- **Body limits**: 1 MB normal, 50 MB on `/research/upload`.
- **SSRF**: webhook URLs are validated against an allowlist.
- **Document upload**: uploaded files are processed in-memory and
  discarded after the run completes; we do not train on user content.
- **Logging**: full request/response is sampled for ops and is purged on
  a 30-day rolling window. No prompts or reports are shared with third
  parties.

## Reporting an incident

If you see a verified-`true` citation that misrepresents the source it
points to — or any other behavior that breaks the contract above —
open an issue at
[github.com/ravidsrk/krawl/issues](https://github.com/ravidsrk/krawl/issues)
or email `team@krawl.sh`. Include the `result_id` and the offending
`[n]`; we'll attach the audit trail.

## Evidence time and access

Research sources carry optional `retrieved_at` (UTC), `published_date`,
`updated_date`, date provenance, `access_status`, `source_record_id` and
`temporal_status`. Missing fields on older reports mean unknown; never assign
export time as retrieval time. Access states are `inspected_text`,
`partial_excerpt`, `provider_summary`, `metadata_only`, `fetch_failed`, or
`unknown`. These describe available evidence, not claim support or truth.
A recent fetch does not make an old article current. Summary/metadata-only
results do not establish that the original text was read.

Optional `source_policy.as_of` sets a UTC evidence cutoff; dates after it are
quarantined in `quarantined_sources`, outside synthesis. Without it the cutoff
is research start. This is not a historical web archive; undated content
remains uncertain. See [evidence methodology](https://docs.krawl.sh/docs/trust#evidence-time-and-access).

Publication/update dates are provider assertions. Exa, Tavily, Serper, Brave,
DuckDuckGo and TinyFish search results are labelled excerpts. Exa/Firecrawl
extractions and capped HTTP pages remain excerpts. The X search overview is
a provider summary; empty linked posts are metadata-only. Native X post bodies
that fit in the content limit are inspected text. Academic abstracts are excerpts;
Semantic Scholar year-only dates keep year precision. arXiv Atom `published`
and `updated` are kept separately. GitHub repository `updated_at` is an update,
not a publication date. Multi-provider crypto summaries do not establish that
the linked page was inspected. Missing provenance stays unknown.

HTML, PDF, DOCX, slides, Markdown and JSON exports include source observation
limits. Retained results store those fields under the same ownership, deletion
and retention rules as the report. ZDR runs do not persist them.

CI uses a synthetic, frozen cutoff cohort (2026-10-01 UTC) with labelled old/new
conflicts, false premises, missing dates and partial X access. It tests propagation
and filtering, not model correctness. No live FreshQA score or improvement is
claimed; a current live cohort needs separately approved budget and labels.

## Evidence coverage

The optional `evidence_coverage` result field gives descriptive counts of unique
exact URLs discovered, explicitly fetched, retained, referenced in outgoing
model input, and cited. Stages overlap: a retained search snippet may never
have had its original page fetched. It also reports explicit failed/unavailable
fetches, unknown fetch/access states, retained-host concentration, missing
publication dates and their known range. A model-input reference may contain
only metadata or a short excerpt; a citation is not a support verdict.

These counts do not establish representative sampling or correct selection
bias. Hostnames do not prove independent origins; mirrors may share a publisher.
Old reports without this field have unknown coverage. No extra LLM call is made;
the measured instrumentation overhead is reported as `measurement_overhead_ms`,
not a guarantee of zero added latency. See [evidence methodology](https://docs.krawl.sh/docs/trust#evidence-coverage).

## Corroboration observations

Learning records may include `corroboration` text-reuse observations. Exact and
near copies form candidate origin groups; these are not proven publishers or
independent sources. Overlapping words can contradict one another. Both
`claim_support` and `independent_origins` remain `unassessed`, and this heuristic
does not increase confidence or reject a unique primary record. Comparison is
bounded to 256 sources and 12,000 characters/1,000 tokens per source; a truncation
flag exposes incomplete comparison. These observations are not a poison detector.

These checks detect possible text reuse only. Factual poisoning and instruction
attacks require separate evaluation. The live paired cohort remains pending.

---


# Batch & Bulk Operations

Source: https://docs.krawl.sh/docs/batch

Krawl supports batch/bulk research three ways:

1. **`POST /research/batch`** — submit up to **20 queries in a single request**.
   Each is queued as an independent background task and you get one task handle
   per query back. This is the formal batch endpoint, defined in
   [`/openapi.json`](https://api.krawl.sh/openapi.json).
2. **Parallel async tasks** — fan out many `POST /research/async` jobs and poll
   `GET /tasks` manually (the lower-level equivalent of the batch endpoint).
3. **`POST /research/compare`** — a multi-result primitive that diffs two prior
   results in a single call (useful for change detection over a batch of saved
   results).

## Batch endpoint — `POST /research/batch`

Submit an array of queries and receive a task handle for each. The server queues
them concurrently (subject to a per-key concurrency cap) and never holds open SSE
connections.

```bash
curl -X POST https://api.krawl.sh/research/batch \
  -H "X-API-Key: krawl_live_..." -H "Content-Type: application/json" \
  -d '{
    "queries": [
      {"query": "State of AI agents 2025", "mode": "deep"},
      {"query": "DeFi TVL trends Q1", "mode": "quick"},
      {"query": "Latest Llama benchmarks", "mode": "deep", "webhook_url": "https://example.com/hook"}
    ]
  }'
```

Response (`202 Accepted`):

```json
{
  "accepted": 3,
  "tasks": [
    {"id": "550e8400-...", "query": "State of AI agents 2025", "mode": "deep", "status": "queued"},
    {"id": "661f9511-...", "query": "DeFi TVL trends Q1", "mode": "quick", "status": "queued"},
    {"id": "772fa622-...", "query": "Latest Llama benchmarks", "mode": "deep", "status": "queued"}
  ]
}
```

| Field | Type | Notes |
|-------|------|-------|
| `queries` | array (1–20) | Each item: `query` (required), `mode` (`deep`/`quick`/`crypto`), optional `webhook_url`. |
| `accepted` | int | Number of jobs queued. |
| `tasks[]` | array | One task handle per query — poll `GET /tasks/{id}`. |

Then poll each task (see [Pattern A](#pattern-a--parallel-async-research)) or use
per-item `webhook_url` callbacks. If the batch would exceed your active-task cap,
the whole request is rejected with `429` + `Retry-After` rather than partially
queued — [retry with backoff](/docs/rate-limits#retry-playbook-exponential-backoff--jitter).

## Pattern A — parallel async research

Submit each query as its own background task, then poll for completion. The
server runs them concurrently (subject to a per-key concurrency cap) and you
never hold open SSE connections.

### 1. Submit jobs

```bash
curl -X POST https://api.krawl.sh/research/async \
  -H "X-API-Key: krawl_live_..." -H "Content-Type: application/json" \
  -d '{"query": "State of AI agents 2025", "mode": "deep"}'
```

```json
{
  "id": "550e8400-...",
  "query": "State of AI agents 2025",
  "mode": "deep",
  "status": "queued",
  "result_id": null,
  "created_at": "2025-01-15T10:00:00Z"
}
```

Repeat for each query in your batch, collecting the returned task `id`s.

**Concurrency cap.** Each API key can only have a bounded number of
queued+running tasks at once. When you exceed it, submission returns `429` with
a message like `Too many active async tasks (N/MAX)`. Throttle submissions and
[retry with backoff](/docs/rate-limits#retry-playbook-exponential-backoff--jitter),
or wait for in-flight tasks to finish.

### 2. Poll for completion

Poll the collection (newest first) or individual tasks:

```bash
# All your tasks
curl -H "X-API-Key: krawl_live_..." "https://api.krawl.sh/tasks?limit=50&offset=0"

# A single task
curl -H "X-API-Key: krawl_live_..." "https://api.krawl.sh/tasks/550e8400-..."
```

Each task has a `status` of `queued`, `running`, `done`, or `error`. When `done`,
fetch the report via `GET /results/{result_id}`:

```json
{
  "id": "550e8400-...",
  "query": "State of AI agents 2025",
  "mode": "deep",
  "status": "done",
  "result_id": "9f1c2b34-...",
  "finished_at": "2025-01-15T10:06:12Z"
}
```

### 3. (Optional) get notified instead of polling

Pass `webhook_url` on submission to receive a `completed`/`error` callback per
task instead of polling. See [Webhooks](/docs/webhooks#async-research-task-events).

### Batch driver (pseudocode)

```python
import time, httpx

queries = ["q1", "q2", "q3", "..."]
client = httpx.Client(base_url="https://api.krawl.sh", headers={"X-API-Key": "krawl_live_..."})

# Submit (respecting the concurrency cap: retry 429s with backoff)
task_ids = []
for q in queries:
    r = client.post("/research/async", json={"query": q, "mode": "deep"})
    r.raise_for_status()
    task_ids.append(r.json()["id"])

# Poll until all terminal
pending = set(task_ids)
while pending:
    for tid in list(pending):
        t = client.get(f"/tasks/{tid}").json()
        if t["status"] in ("done", "error"):
            pending.discard(tid)
            if t["status"] == "done":
                report = client.get(f"/results/{t['result_id']}").json()
                # ... use report
    time.sleep(5)  # poll interval; honor rate-limit headers
```

> Use a polling interval (e.g. 5s) and watch `X-RateLimit-Remaining` so polling
> a large batch doesn't trip the rate limiter. See [Rate Limits & Retries](/docs/rate-limits).

## Pattern B — `POST /research/compare`

`compare` is the multi-result batch primitive for **change detection**: it takes
results you already have and returns a structured diff in one call.

**Request** (`CompareRequest`):

| Field | Type | Description |
|-------|------|-------------|
| `query` | string | *required* — query to find/compare results for (1–2000 chars) |
| `result_ids` | string[] | optional, up to 10 — specific results to compare |

Behavior depends on how many `result_ids` you pass:

- **2+ ids** → compares the first two of those results directly.
- **1 id** → compares it against the most similar previous result for its query.
- **0 ids** → finds the two most recent results matching `query` and compares them.

```bash
curl -X POST https://api.krawl.sh/research/compare \
  -H "X-API-Key: krawl_live_..." -H "Content-Type: application/json" \
  -d '{"query": "AI agents", "result_ids": ["9f1c...", "7b3e..."]}'
```

**Response:**

```json
{
  "old_result_id": "7b3e...",
  "new_result_id": "9f1c...",
  "old_query": "AI agents",
  "new_query": "AI agents",
  "old_created_at": "2025-01-08T10:00:00Z",
  "new_created_at": "2025-01-15T10:00:00Z",
  "changes": [{ "field": "...", "old": "...", "new": "..." }],
  "new_findings": ["..."],
  "removed_claims": ["..."],
  "summary": "Natural-language summary of what changed.",
  "has_changes": true
}
```

If fewer than two comparable results exist, `compare` returns `404` with a
message telling you how many it found — submit more research first.

## Choosing a pattern

| Goal | Use |
|------|-----|
| Run many independent queries at once | **Pattern A** (parallel `/research/async` + `/tasks`) |
| Detect what changed between two reports | **Pattern B** (`/research/compare`) |
| Recurring watch + auto-diff on a schedule | [Lookouts](/docs/lookouts) |

---


# Webhooks

Source: https://docs.krawl.sh/docs/webhooks

Krawl delivers **outbound webhooks** so agents don't have to hold an SSE
connection or poll. Two features emit them:

- **Async research** ([`POST /research/async`](/docs/api-reference)) — pass a
  `webhook_url` and Krawl notifies you when the background task finishes (or fails).
- **Scheduled lookouts** ([Lookouts](/docs/lookouts)) — recurring research jobs
  fire webhooks at each lifecycle stage: `started`, `progress`, `completed`, `error`.

Webhooks are delivered as an HTTP `POST` with a JSON body to the `webhook_url`
you registered on the resource. Delivery is **fire-and-forget**: Krawl does not
currently retry failed deliveries, so your endpoint should respond quickly
(within ~10s) and `2xx`. Treat webhooks as an optimization on top of the
authoritative state you can always fetch via `GET /tasks/{id}` (async) or
`GET /lookouts/{id}` + `GET /results/{id}` (lookouts).

> **Outbound vs. inbound.** This page documents Krawl's *own* outbound webhooks.
> The separate Stripe billing webhook (`POST /billing/webhook`) is *inbound* — it
> receives events *from* Stripe and verifies Stripe's own signature. That is
> unrelated to the signing scheme below.

## Event payloads

Every webhook body is a JSON object with an `event` field and a `timestamp`
(RFC 3339 / ISO 8601, UTC). The other fields depend on the source.

### Async research task events

Registered via `webhook_url` on `POST /research/async`.

**`completed`** — the background task finished and a result was saved:

```json
{
  "event": "completed",
  "task_id": "550e8400-e29b-41d4-a716-446655440000",
  "result_id": "9f1c2b34-...",
  "query": "Latest developments in AI agents",
  "timestamp": "2025-01-15T10:05:32.184Z"
}
```

**`error`** — the task failed (or the server had no capacity to start it):

```json
{
  "event": "error",
  "task_id": "550e8400-e29b-41d4-a716-446655440000",
  "error": "server_busy",
  "timestamp": "2025-01-15T10:05:32.184Z"
}
```

Fetch the full report with `GET /results/{result_id}` once you receive a
`completed` event, or re-check status with `GET /tasks/{task_id}`.

### Lookout events

Registered via `webhook_url` on `POST /lookouts`. A single run emits these in
order: `started` → `progress` (repeating) → `completed` (or `error`).

**`started`**:

```json
{
  "event": "started",
  "lookout_id": "7b3e...",
  "query": "Latest developments in AI agents",
  "timestamp": "2025-01-15T10:00:00.000Z"
}
```

**`progress`** (emitted roughly every 5 research steps):

```json
{
  "event": "progress",
  "lookout_id": "7b3e...",
  "query": "Latest developments in AI agents",
  "steps_completed": 5,
  "timestamp": "2025-01-15T10:01:12.000Z"
}
```

**`completed`** (includes the change diff vs. the previous run):

```json
{
  "event": "completed",
  "lookout_id": "7b3e...",
  "query": "Latest developments in AI agents",
  "result_id": "9f1c2b34-...",
  "has_changes": true,
  "diff": {
    "has_changes": true,
    "summary": "Two new model releases since the last run.",
    "new_findings": ["..."],
    "removed_claims": ["..."],
    "changed_data": [{ "field": "...", "old": "...", "new": "..." }]
  },
  "result_preview": "First 500 characters of the report markdown...",
  "timestamp": "2025-01-15T10:05:00.000Z"
}
```

**`error`**:

```json
{
  "event": "error",
  "lookout_id": "7b3e...",
  "query": "Latest developments in AI agents",
  "error": "research engine timeout",
  "steps_completed": 12,
  "timestamp": "2025-01-15T10:05:00.000Z"
}
```

## Signature verification

> **Rollout status.** Signed delivery is the documented contract for outbound
> webhooks. When signing is enabled for your endpoint, every delivery carries an
> `X-Krawl-Signature` header computed as below. Endpoints registered before
> signing was enabled may receive unsigned deliveries during the transition —
> verify the signature when the header is present, and plan to require it.

Krawl signs each webhook using **HMAC-SHA256** over the **raw request body**
with a **per-endpoint signing secret** (`whsec_...`). The signature is sent in
the `X-Krawl-Signature` header using a Stripe-style scheme that also carries the
signed timestamp:

```
X-Krawl-Signature: t=1736935500,v1=5257a869e7 ... 8e1f9c
```

- `t` — Unix timestamp (seconds) when the signature was generated.
- `v1` — lowercase hex HMAC-SHA256 of the string `"{t}.{raw_body}"`, keyed with
  your endpoint's signing secret.

The **signed payload** is the timestamp, a literal `.`, and the exact raw bytes
of the request body:

```
signed_payload = "{t}" + "." + raw_request_body
expected_v1    = hex( HMAC_SHA256(signing_secret, signed_payload) )
```

To verify: recompute `expected_v1` and compare it to the `v1` value using a
**constant-time** comparison. Reject the request if they differ.

### Node.js

```js
import crypto from "node:crypto";

// rawBody MUST be the exact bytes received (do not re-serialize parsed JSON).
function verifyKrawlWebhook(rawBody, signatureHeader, signingSecret, toleranceSec = 300) {
  const parts = Object.fromEntries(
    signatureHeader.split(",").map((kv) => kv.split("=")),
  );
  const t = Number(parts.t);
  const v1 = parts.v1;
  if (!t || !v1) throw new Error("malformed signature header");

  // Replay protection: reject stale timestamps.
  const ageSec = Math.abs(Date.now() / 1000 - t);
  if (ageSec > toleranceSec) throw new Error("timestamp outside tolerance");

  const expected = crypto
    .createHmac("sha256", signingSecret)
    .update(`${t}.${rawBody}`, "utf8")
    .digest("hex");

  const ok = crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(v1));
  if (!ok) throw new Error("signature mismatch");
  return true;
}

// Express example — capture the raw body:
// app.use("/webhook", express.raw({ type: "application/json" }));
app.post("/webhook", (req, res) => {
  try {
    verifyKrawlWebhook(req.body.toString("utf8"), req.get("X-Krawl-Signature"), process.env.KRAWL_WEBHOOK_SECRET);
  } catch (e) {
    return res.status(400).send(`invalid signature: ${e.message}`);
  }
  const event = JSON.parse(req.body.toString("utf8"));
  // ... handle event.event ("completed", "error", ...)
  res.sendStatus(200);
});
```

### Python

```python
import hmac
import hashlib
import time


def verify_krawl_webhook(raw_body: bytes, signature_header: str, signing_secret: str, tolerance_sec: int = 300) -> bool:
    parts = dict(kv.split("=", 1) for kv in signature_header.split(","))
    t = int(parts["t"])
    v1 = parts["v1"]

    # Replay protection: reject stale timestamps.
    if abs(time.time() - t) > tolerance_sec:
        raise ValueError("timestamp outside tolerance")

    signed_payload = f"{t}.".encode() + raw_body
    expected = hmac.new(signing_secret.encode(), signed_payload, hashlib.sha256).hexdigest()

    if not hmac.compare_digest(expected, v1):
        raise ValueError("signature mismatch")
    return True


# FastAPI example:
# @app.post("/webhook")
# async def webhook(request: Request):
#     raw = await request.body()
#     verify_krawl_webhook(raw, request.headers["X-Krawl-Signature"], os.environ["KRAWL_WEBHOOK_SECRET"])
#     event = json.loads(raw)
#     ...
```

### Verifying manually with curl / shell

You can reproduce the `v1` value from a captured body and timestamp:

```bash
# t and raw body must match exactly what was delivered.
T=1736935500
BODY='{"event":"completed","task_id":"550e8400-...","result_id":"9f1c...","query":"...","timestamp":"2025-01-15T10:05:32.184Z"}'

printf '%s.%s' "$T" "$BODY" \
  | openssl dgst -sha256 -hmac "$KRAWL_WEBHOOK_SECRET" -hex \
  | sed 's/^.* //'
# Compare the output to the v1=... value in the X-Krawl-Signature header.
```

## Replay protection

The signed timestamp (`t`) defends against replay attacks:

1. **Enforce a tolerance window** (the examples use 300s / 5 minutes). Reject any
   delivery whose `t` is too far from your server clock — a captured-and-replayed
   request will eventually fall outside the window.
2. Because `t` is part of the signed payload, an attacker cannot rewrite the
   timestamp without invalidating `v1`.
3. For exactly-once processing, also **deduplicate** on the event's natural key
   (`task_id` / `lookout_id` + `event`), since retried or duplicated deliveries
   can occur.

## Best practices

- **Always read the raw body** before parsing JSON — re-serializing changes bytes
  and breaks the HMAC.
- **Respond fast (`2xx`) and process async.** Krawl uses a ~10s delivery timeout
  and does not retry; do heavy work after acknowledging.
- **Fall back to polling.** Webhooks are best-effort. The source of truth is
  `GET /tasks/{id}` (async) and `GET /lookouts/{id}` + `GET /results/{id}`
  (lookouts). See [Rate Limits & Retries](/docs/rate-limits) for backoff guidance.
- **Use HTTPS endpoints.** Krawl validates webhook URLs against SSRF rules and
  will refuse to deliver to private/internal addresses.

---


# Rate Limits & Retries

Source: https://docs.krawl.sh/docs/rate-limits

Krawl rate-limits requests per plan (and per IP for anonymous/demo traffic).
This page documents the response headers that advertise your current limit
state, the structured error envelope returned on failure, and a concrete
retry/backoff playbook agents should implement.

## Plan limits

| Plan | Rate limit | Notes |
|------|-----------|-------|
| `demo` | ~5 / hour | 24h key, evaluation only |
| `free` | 10 / min | |
| `payg` | 60 / min | metered, pay-per-request |
| `pro` | 120 / min | |

Anonymous/unauthenticated traffic defaults to **5 requests/minute per IP**. Some
endpoints set their own tighter limits (e.g. manual lookout triggers at 2/min).

Separately, async fan-out is bounded: `POST /research/async` rejects new
submissions with `429` once you have too many concurrent queued+running tasks.

## Anonymous playground (`POST /playground`)

The homepage demo at `krawl.sh` runs against `POST /playground`, a no-key,
no-signup surface. Because it is unauthenticated it gets its own throttle layer
on top of the global anonymous limit:

| Knob | Default | Notes |
|------|---------|-------|
| Per-IP cap | **15 / hour** | Hard ceiling — raised from `3/hour` (2026-06) to fit a small team on a corporate NAT. |
| Per-session priority slot | **3 / hour** | A first-time visitor's first 3 runs bypass the IP cap (see below). |
| Per-run cost ceiling | `PLAYGROUND_MAX_COST_USD` ($0.15) | Bounds spend per run. |
| Query length | `PLAYGROUND_MAX_QUERY_LEN` (300 chars) | Shorter than the 2000-char API cap. |
| Mode | forced `quick` | Cheapest, fastest path. |

**The session-priority trade-off.** On a shared egress IP (corporate NAT, café,
co-working space) a strict IP-only cap penalises the *first* visitor for traffic
their colleague behind the same NAT has already used. The priority slot avoids
that: each fresh browser cookie (`krawl_pg_session`, HttpOnly, lax, 1h TTL,
scoped to `/playground`) gets **3 priority runs that bypass the per-IP 15/hour
cap**. After those 3 runs are spent, subsequent runs from the same cookie fall
back to the IP cap.

Two things to know:

- **Clearing cookies does not grant new slots beyond the IP cap.** A
  freshly-minted cookie still consults the IP cap on its first run — the
  priority slot only protects *returning visitors with priority left*, not
  anonymous cookie-rotators. This is what makes the per-IP `15/hour` a hard
  ceiling.
- **The counter is process-local.** Every Fly auto-suspend or machine restart
  resets the playground's per-IP and per-session windows. This is acceptable for
  a marketing surface (worst case is one extra request per restart) and keeps
  the playground from depending on Postgres.

If you need higher limits, [get an API key](/docs/getting-started) or pay
per-request via [x402](/docs/payments-x402); the playground is an activation
surface, not a free API tier.

## Rate-limit headers

Responses carry standard rate-limit headers so agents can self-throttle **before**
hitting a `429`:

| Header | Meaning |
|--------|---------|
| `X-RateLimit-Limit` | Max requests allowed in the current window |
| `X-RateLimit-Remaining` | Requests left in the current window |
| `X-RateLimit-Reset` | Unix timestamp (seconds) when the window resets |
| `Retry-After` | **On `429` only** — seconds to wait before retrying |

Read `X-RateLimit-Remaining` on every response. When it approaches `0`, pace
yourself until `X-RateLimit-Reset` rather than firing requests into a `429`.

## The 429 response

When you exceed the limit, Krawl returns HTTP `429` with the standard error
envelope and a `Retry-After` header:

```http
HTTP/1.1 429 Too Many Requests
Retry-After: 60
X-RateLimit-Limit: 10
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1736935560
Content-Type: application/json

{
  "error_code": "RATE_LIMITED",
  "message": "Rate limit exceeded",
  "detail": "Rate limit exceeded",
  "retry_after": 60
}
```

## Error envelope

All errors share one JSON shape so agents can parse failures uniformly and
decide whether a retry is warranted:

```json
{
  "error_code": "RATE_LIMITED",
  "message": "human-readable message",
  "detail": "...",
  "retry_after": 60
}
```

| Field | Type | Description |
|-------|------|-------------|
| `error_code` | string | Machine-readable code (see below) |
| `message` | string | Human-readable explanation |
| `detail` | string \| null | Extra context (may duplicate `message`) |
| `retry_after` | number \| null | Seconds to wait before retrying, when applicable |

### Error codes & retry eligibility

| `error_code` | HTTP | Retry? | Guidance |
|--------------|------|--------|----------|
| `RATE_LIMITED` | 429 | ✅ Yes | Wait `retry_after` / `Retry-After`, then retry with backoff |
| `INTERNAL_ERROR` | 500 | ✅ Yes | Transient — retry with exponential backoff + jitter |
| `quota_exceeded` | 402 | ❌ No | Monthly plan quota exhausted — upgrade or wait for reset |
| `VALIDATION_ERROR` | 422 | ❌ No | Fix the request; retrying the same body will fail again |
| `NOT_FOUND` | 404 | ❌ No | Resource doesn't exist; don't retry |

**Rule of thumb:** retry on `429` and `5xx`. Do **not** retry `4xx` other than
`429` — those indicate a client-side problem the same request won't fix. A `402`
means payment/quota is required (see [Payments & x402](/docs/payments-x402)).

## Retry playbook (exponential backoff + jitter)

Agents should implement bounded exponential backoff with full jitter, and
**honor `Retry-After`/`retry_after` when present** (it overrides your computed
delay).

```python
import random
import time
import httpx

RETRYABLE_STATUS = {429, 500, 502, 503, 504}


def request_with_retry(client, method, url, *, max_retries=5, base=0.5, cap=30.0, **kwargs):
    for attempt in range(max_retries + 1):
        resp = client.request(method, url, **kwargs)
        if resp.status_code not in RETRYABLE_STATUS:
            return resp  # success or non-retryable error
        if attempt == max_retries:
            return resp  # give up, let caller handle

        # Prefer the server's hint when present.
        retry_after = resp.headers.get("Retry-After")
        if retry_after is not None:
            delay = float(retry_after)
        else:
            # Exponential backoff with full jitter.
            delay = random.uniform(0, min(cap, base * (2 ** attempt)))
        time.sleep(delay)
    return resp


with httpx.Client(base_url="https://api.krawl.sh", headers={"X-API-Key": "krawl_live_..."}) as c:
    r = request_with_retry(c, "POST", "/research", json={"query": "What is EIP-4844?", "mode": "quick"})
```

```js
const RETRYABLE = new Set([429, 500, 502, 503, 504]);

async function requestWithRetry(url, init, { maxRetries = 5, base = 500, cap = 30000 } = {}) {
  for (let attempt = 0; ; attempt++) {
    const resp = await fetch(url, init);
    if (!RETRYABLE.has(resp.status) || attempt === maxRetries) return resp;

    const retryAfter = resp.headers.get("Retry-After");
    const delay = retryAfter
      ? Number(retryAfter) * 1000
      : Math.random() * Math.min(cap, base * 2 ** attempt); // full jitter
    await new Promise((r) => setTimeout(r, delay));
  }
}
```

### Guidelines

- **Cap retries** (e.g. 5) and **cap the delay** (e.g. 30s) so a stuck agent
  doesn't loop forever.
- **Always add jitter** — synchronized retries from a fleet of agents cause
  thundering-herd spikes that re-trigger `429`s.
- **Honor the server hint.** If `Retry-After` or `retry_after` is set, wait at
  least that long.
- **Self-throttle proactively** using `X-RateLimit-Remaining` / `X-RateLimit-Reset`
  to avoid `429`s in the first place.
- For long-running work, prefer [`POST /research/async`](/docs/api-reference) and
  poll `GET /tasks/{id}` instead of holding a connection — it spreads load and
  avoids retry storms. See [Status & Uptime](/docs/status) for the full recovery
  playbook.

---


# API Versioning & Deprecation

Source: https://docs.krawl.sh/docs/versioning

Krawl uses **URL-based versioning** and a predictable deprecation policy so
agents can depend on stable behavior and get machine-readable advance notice
before anything is removed.

## URL versioning

Versioned, compatibility-sensitive endpoints live under a version prefix:

```
https://api.krawl.sh/v1/chat/completions
https://api.krawl.sh/v1/models
```

- The `/v1/*` namespace is the **stable, OpenAI-compatible** surface. Breaking
  changes to these shapes ship under a new prefix (`/v2/...`), never in place.
- Some core endpoints (e.g. `/research`, `/tasks`, `/lookouts`) are currently
  served unversioned at the root. Treat the unversioned root as the current
  major version; additive, backward-compatible changes (new optional fields, new
  endpoints) can land there without a version bump.

**Backward-compatible changes** we may make without a version change:

- Adding new endpoints or new optional request fields.
- Adding new fields to response objects.
- Adding new enum values (e.g. new research `mode`s or `source` types).

Build clients to **tolerate unknown fields** and ignore enum values they don't
recognize.

## Deprecation policy

When an endpoint or field is scheduled for removal:

1. **Minimum 6-month notice window** between the deprecation announcement and the
   sunset (removal) date.
2. Deprecated responses carry standard **RFC 8594** headers so agents detect the
   deprecation programmatically — no need to read a changelog.
3. A migration guide is linked directly from the response.

### Deprecation headers (RFC 8594)

A deprecated endpoint returns:

```http
HTTP/1.1 200 OK
Deprecation: Wed, 01 Oct 2025 00:00:00 GMT
Sunset: Sun, 01 Mar 2026 00:00:00 GMT
Link: <https://docs.krawl.sh/docs/versioning>; rel="deprecation"; type="text/html"
```

| Header | Meaning |
|--------|---------|
| `Deprecation` | The date the endpoint became deprecated (HTTP date). It may also appear as `Deprecation: true`. |
| `Sunset` | The date after which the endpoint may stop working (HTTP date, RFC 8594). At least 6 months after `Deprecation`. |
| `Link … rel="deprecation"` | URL of the migration documentation for this endpoint. |

### What agents should do

- **Log a warning** whenever a response contains a `Deprecation` or `Sunset`
  header, including the endpoint and the `Sunset` date.
- **Schedule migration** before the `Sunset` date — after it, the endpoint may
  return `404`/`410` or redirect.
- **Follow the `rel="deprecation"` link** for the replacement endpoint and
  migration steps.

```python
resp = client.get("/v1/models")
if "Sunset" in resp.headers:
    logger.warning(
        "Endpoint deprecated; sunset on %s — migrate per %s",
        resp.headers["Sunset"],
        resp.headers.get("Link", "https://docs.krawl.sh/docs/versioning"),
    )
```

## Checking for deprecations

You can proactively detect deprecations on any call by inspecting response
headers — there is no separate endpoint required. Combine this with the
[OpenAPI spec](https://api.krawl.sh/openapi.json), which always reflects the
current, supported surface.

---


# Status & Uptime

Source: https://docs.krawl.sh/docs/status

This page tells agents how to check whether Krawl is healthy and how to recover
gracefully when it isn't.

## Service status

- **Live status page (human):** [https://docs.krawl.sh/status](/status) —
  real-time component status, API version, uptime, and probe latency (probes the
  API health endpoint on every load).
- **Status JSON (programmatic):**
  [`https://docs.krawl.sh/status.json`](/status.json) — machine-readable status
  with per-component state, so agents can determine retry eligibility.
- **Health endpoint (programmatic):**
  [`https://api.krawl.sh/health`](https://api.krawl.sh/health) — a fast,
  unauthenticated liveness/readiness check suitable for automated probes.

## What the status probe establishes

The page and JSON endpoint use the same bounded health probe. A successful HTTP
status alone is insufficient: both `status: "ok"` and `db: "ok"` must be present
to show operational. HTTP errors and degraded payloads show degradation; invalid
or missing payloads show unknown. Only allowlisted health fields are exposed;
raw errors and arbitrary upstream fields are discarded.

MCP, research providers, payment settlement and webhooks are not independently
probed by `/health`. MCP is therefore **unknown**, even while API health is
green. There is no incident timeline or historical availability measurement.
API uptime is the current process uptime, not a measured service SLO. The
existing page refreshes on request and does not create new status infrastructure.

## The `/health` endpoint

`GET /health` requires no auth and reports liveness, version, and database
connectivity:

```bash
curl https://api.krawl.sh/health
```

```json
{
  "status": "ok",
  "version": "0.2.0",
  "db": "ok",
  "uptime_seconds": 1234.5
}
```

| Field | Meaning |
|-------|---------|
| `status` | `ok` when the database is reachable, otherwise `degraded` |
| `db` | `ok` or `error` |
| `version` | Running API version |
| `uptime_seconds` | Process uptime |

**Agent usage:** Before a batch of work — or as a circuit-breaker probe after
repeated failures — call `/health`. If `status` is `degraded` or the request
fails/times out, back off and retry the probe rather than hammering the research
endpoints. Keep probes lightweight (e.g. every 30–60s while degraded).

## Error-recovery playbook

When a request fails, decide your action from the HTTP status and the
[error envelope](/docs/rate-limits#error-envelope) (`{error_code, message, retry_after}`):

| Situation | HTTP | Action |
|-----------|------|--------|
| Rate limited | `429` | Wait `Retry-After` / `retry_after`, retry with backoff |
| Server/transient error | `5xx` | Retry with exponential backoff + jitter |
| Quota exhausted | `402` `quota_exceeded` | Stop; upgrade plan or wait for monthly reset |
| Bad request | `4xx` (not 429) | Fix the request; do not retry as-is |
| Repeated `5xx` / timeouts | — | Open a local circuit breaker; probe `/health` before resuming |

Recommended loop for autonomous agents:

1. **Probe `/health`** if you're starting fresh or recovering from a failure streak.
2. **Send the request**, honoring `X-RateLimit-Remaining` to self-throttle.
3. **On retryable failure** (`429`/`5xx`), back off (exponential + jitter, capped),
   preferring the server's `Retry-After` hint. See the full
   [retry playbook](/docs/rate-limits#retry-playbook-exponential-backoff--jitter).
4. **On a failure streak**, open a circuit breaker: pause requests, poll `/health`
   until `status: ok`, then resume.
5. **For long jobs**, use [`POST /research/async`](/docs/api-reference) +
   `GET /tasks/{id}` polling (or [webhooks](/docs/webhooks)) so a dropped
   connection never loses work — the task keeps running server-side and the
   result is retrievable via `GET /results/{id}`.

## Idempotency & not losing work

- Async tasks and lookouts run server-side. If your client crashes, the work
  continues; re-fetch state with `GET /tasks/{id}` or `GET /lookouts/{id}`.
- Webhook deliveries are best-effort and may be duplicated — **deduplicate** on
  `task_id`/`lookout_id` + `event`. See [Webhooks](/docs/webhooks#replay-protection).

---


# Crypto Research Tools

Source: https://docs.krawl.sh/docs/crypto-tools

Krawl includes 11 specialized crypto research tools that aggregate data from professional-grade APIs. These tools are automatically available when using crypto research modes, and can be mixed with standard web/X/GitHub sources.

## Tools Overview

| Tool | Data Source | Key Output |
|---|---|---|
| `cryptoData` | CoinGecko | Price, market cap, volume, supply, ATH |
| `cryptoChart` | CoinGecko | Historical price data for charting |
| `defiRanking` | DeFi Llama | Protocol TVL rankings, chain TVLs |
| `onchainAnalytics` | Nansen, Dune | Active addresses, tx count, gas, bridge flows |
| `protocolDeepDive` | DeFi Llama + web | Deep protocol analysis with TVL breakdown |
| `tokenRiskAssessment` | Multiple | Audit history, exploits, smart contract risks |
| `smartMoneyTracker` | Nansen, Coinglass | Whale flows, institutional positioning |
| `narrativeTracker` | LunarCrush | Social sentiment, Galaxy Score, engagement |
| `derivativesAnalysis` | Coinglass | Funding rates, OI, liquidations, perps data |
| `yieldFinder` | DeFi Llama | APY/APR across protocols and chains |
| `governanceMonitor` | Messari, web | DAO proposals, voting activity, governance |

## API Keys

| Provider | Env Variable | Required |
|---|---|---|
| CoinGecko Pro | `COINGECKO_API_KEY` | **Yes** (core token data) |
| Nansen | `NANSEN_API_KEY` | Optional |
| Dune | `DUNE_API_KEY` | Optional |
| LunarCrush | `LUNARCRUSH_API_KEY` | Optional |
| Coinglass | `COINGLASS_API_KEY` | Optional |
| Messari | `MESSARI_API_KEY` | Optional |

DeFi Llama is free and requires no API key.

## Crypto Research Modes

Five research modes are crypto-optimized:

### `crypto`
General crypto research. All crypto tools available alongside web and X search.

```bash
curl -N -X POST https://api.krawl.sh/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"query": "State of DeFi in 2025", "mode": "crypto"}'
```

### `token-analysis`
Single token deep dive. Prioritizes `cryptoData`, `cryptoChart`, `tokenRiskAssessment`, and `smartMoneyTracker`.

### `protocol-research`
DeFi protocol analysis. Prioritizes `defiRanking`, `protocolDeepDive`, `governanceMonitor`.

### `whale-tracking`
Smart money flows. Prioritizes `smartMoneyTracker`, `onchainAnalytics`, `derivativesAnalysis`.

### `risk-assessment`
Security and risk. Prioritizes `tokenRiskAssessment`, audit history, exploit research.

### `yield-strategy`
Yield optimization. Prioritizes `yieldFinder`, `defiRanking`, protocol comparison.

### `narrative`
Market narratives and social sentiment. Prioritizes `narrativeTracker`, X/Twitter search, community sources.

## Tool Details

### `cryptoData`

Fetches current token data from CoinGecko Pro API.

**Returns:** price, market cap, 24h volume, price change, circulating/total supply, ATH, ATH date

**Agent trigger keywords:** "price", "market cap", "volume", "token", "coin", "supply"

### `cryptoChart`

Fetches historical price data for charting.

**Returns:** Time series data suitable for candlestick or line charts

**Agent trigger keywords:** "chart", "price history", "trend", "performance over"

### `defiRanking`

Protocol TVL rankings from DeFi Llama.

**Returns:** Protocol name, TVL, chain breakdown, percentage changes

**Agent trigger keywords:** "tvl", "total value locked", "protocol rank", "defi ranking"

### `onchainAnalytics`

On-chain metrics from Nansen and Dune Analytics.

**Returns:** Daily active addresses, transaction counts, DEX volume, gas fees, bridge flows

**Agent trigger keywords:** "on-chain", "wallet activity", "daily active", "dex volume", "gas fee"

### `protocolDeepDive`

Deep analysis of a specific DeFi protocol.

**Returns:** TVL breakdown by chain, revenue metrics, user growth, competitive position

**Agent trigger keywords:** "tvl breakdown", "protocol analysis", "deep dive"

### `tokenRiskAssessment`

Security and risk analysis.

**Returns:** Audit history, known vulnerabilities, bug bounty status, admin key risks

**Agent trigger keywords:** "risk", "audit", "exploit", "vulnerability", "security"

### `smartMoneyTracker`

Whale and institutional flow tracking via Nansen.

**Returns:** Smart money inflows/outflows, whale wallet activity, institutional positions

**Agent trigger keywords:** "smart money", "whale", "institutional", "inflow", "outflow"

### `narrativeTracker`

Social sentiment analysis via LunarCrush.

**Returns:** Galaxy Score, social volume, engagement metrics, sentiment breakdown

**Agent trigger keywords:** "sentiment", "social", "narrative", "community", "galaxy score"

### `derivativesAnalysis`

Derivatives market data from Coinglass.

**Returns:** Funding rates, open interest, liquidation data, futures volume

**Agent trigger keywords:** "funding rate", "open interest", "liquidation", "futures", "perp"

### `yieldFinder`

Yield farming opportunities.

**Returns:** APY/APR by protocol and chain, TVL, risk indicators

**Agent trigger keywords:** "yield", "apy", "apr", "farming", "lending rate"

### `governanceMonitor`

DAO governance tracking via Messari.

**Returns:** Active proposals, voting results, governance participation rates

**Agent trigger keywords:** "governance", "proposal", "vote", "dao", "snapshot"

## Using Crypto Search Directly

For quick token lookups without the full research pipeline:

```bash
# Token data
curl -X POST https://api.krawl.sh/search \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"query": "ethereum", "source": "crypto", "options": {"token_id": "ethereum"}}'

# Search by name
curl -X POST https://api.krawl.sh/search \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"query": "solana", "source": "crypto"}'
```

## Intelligent Tool Routing

The research engine automatically routes queries to the best crypto tool based on keyword analysis. For example:

- "$BTC sentiment" → `xCashtagSearch` (cashtag takes priority)
- "Ethereum market cap" → `cryptoData`
- "Uniswap vs Sushiswap" → `webSearch` (comparison queries need articles)
- "DeFi TVL rankings" → `defiRanking`
- "Whale wallet activity on-chain" → `onchainAnalytics`

---


# Research Templates

Source: https://docs.krawl.sh/docs/templates

Krawl includes 8 research templates — pre-built query patterns with variable substitution. Templates specify the query structure, research mode, and required variables.

## List Templates

```bash
curl -H "X-API-Key: your-key" https://api.krawl.sh/templates
```

## Execute a Template

```bash
curl -N -X POST https://api.krawl.sh/templates/token-due-diligence/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"variables": {"token": "Ethereum"}}'
```

Returns an SSE stream (same as `/research`).

## Available Templates

### `token-due-diligence`

Comprehensive analysis of a crypto token.

**Variables:** `token` (token name or ticker)
**Mode:** `deep`
**Tags:** crypto, investment

**Generated query:**
> Perform due diligence on \{token\}. Cover: tokenomics, team background, technology, competitive landscape, on-chain metrics, community sentiment, risk factors, and investment thesis.

```bash
curl -N -X POST https://api.krawl.sh/templates/token-due-diligence/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"variables": {"token": "SOL"}}'
```

### `protocol-comparison`

Compare two DeFi protocols.

**Variables:** `protocol_a`, `protocol_b`
**Mode:** `deep`
**Tags:** crypto, defi

**Generated query:**
> Compare \{protocol_a\} vs \{protocol_b\}. Cover: TVL, revenue, user growth, security track record, governance, ecosystem, developer activity, and competitive moat.

```bash
curl -N -X POST https://api.krawl.sh/templates/protocol-comparison/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"variables": {"protocol_a": "Uniswap", "protocol_b": "Curve"}}'
```

### `market-narrative`

Analyze an emerging market narrative.

**Variables:** `narrative`
**Mode:** `deep`
**Tags:** crypto, market

**Generated query:**
> Research the \{narrative\} narrative. Cover: key projects, catalysts, timeline, risks, social sentiment on X/Twitter and Reddit, smart money positioning, and potential impact on the broader market.

```bash
curl -N -X POST https://api.krawl.sh/templates/market-narrative/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"variables": {"narrative": "AI x Crypto"}}'
```

### `security-audit-review`

Review security posture of a protocol/project.

**Variables:** `project`
**Mode:** `deep`
**Tags:** crypto, security

**Generated query:**
> Review the security posture of \{project\}. Cover: audit history, known vulnerabilities, bug bounty program, smart contract architecture, admin key risks, upgrade mechanisms, and incident response history.

### `company-research`

Deep dive on a company.

**Variables:** `company`
**Mode:** `deep`
**Tags:** business

**Generated query:**
> Research \{company\} comprehensively. Cover: founding story, business model, revenue, funding, leadership team, competitive landscape, product roadmap, market position, strengths, weaknesses, and outlook.

```bash
curl -N -X POST https://api.krawl.sh/templates/company-research/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"variables": {"company": "Anthropic"}}'
```

### `technology-landscape`

Map a technology category.

**Variables:** `technology`, `year` (default: "2026")
**Mode:** `deep`
**Tags:** technology

**Generated query:**
> Map the \{technology\} landscape in \{year\}. Cover: key players, open source projects, adoption trends, technical architecture patterns, pros/cons of different approaches, and where the field is heading.

### `quick-summary`

Fast overview of any topic.

**Variables:** `topic`
**Mode:** `quick`
**Tags:** general

**Generated query:**
> Give me a comprehensive overview of \{topic\}. Cover the key facts, recent developments, and what matters most.

```bash
curl -N -X POST https://api.krawl.sh/templates/quick-summary/research \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{"variables": {"topic": "MCP protocol"}}'
```

### `person-profile`

Research a public figure.

**Variables:** `person`
**Mode:** `deep`
**Tags:** people

**Generated query:**
> Research \{person\}. Cover: background, career history, notable achievements, public positions, social media presence, controversies, and current activities.

## Template Structure

Each template has:

| Field | Type | Description |
|---|---|---|
| `id` | string | Unique identifier |
| `name` | string | Human-readable name |
| `description` | string | Short description |
| `query_template` | string | Template string with `{variable}` placeholders |
| `mode` | string | Research mode |
| `variables` | array | Variable definitions with `name`, `label`, `placeholder`, optional `default` |
| `tags` | string[] | Categorization tags |

## Variable Substitution

Missing variables use their `default` value if defined. If no default exists and the variable is not provided, the API returns a 422 error.

```json
{
  "variables": {
    "technology": "ZK proofs"
  }
}
```

The `year` variable in `technology-landscape` has a default of "2026", so it's optional.

---


# Memory & Knowledge Base

Source: https://docs.krawl.sh/docs/memory

Krawl maintains a persistent knowledge base across research sessions. Key findings from each research run are automatically extracted and stored, then recalled at the start of future research to avoid re-researching known facts.

## How It Works

### Automatic Memorization (Post-Research)

After every research session completes:

1. The report is sent to **Haiku 4.5** (Tier 1, cheap/fast)
2. The model extracts **5–15 key facts/findings** as structured learnings
3. Each learning gets content, topic tags, source query, and source URLs
4. Learnings are stored in PostgreSQL, scoped to the API key

Only specific, factual, reusable knowledge is extracted — not vague statements or opinions. Each learning must be self-contained (understandable without the original report context).

### Automatic Recall (Pre-Research)

At the start of each research session:

1. The query is searched against the knowledge base (up to 15 results)
2. Matching memories are formatted into a "What you already know" context block
3. This context is injected into the research prompts

The recall prompt instructs the research engine to **avoid re-researching known facts** and focus on finding new information, updates, or details.

## API Endpoints

### Search Memories

```bash
curl -H "X-API-Key: your-key" \
  "https://api.krawl.sh/memory?q=ethereum%20merge&limit=10"
```

**Response:**
```json
{
  "query": "ethereum merge",
  "count": 3,
  "memories": [
    {
      "id": "550e8400-...",
      "content": "Ethereum transitioned from Proof of Work to Proof of Stake on September 15, 2022, reducing energy consumption by ~99.95%.",
      "source_query": "What happened with the Ethereum merge?",
      "source_urls": ["https://ethereum.org/en/upgrades/merge/"],
      "tags": ["ethereum", "proof-of-stake", "merge"],
      "created_at": "2025-01-10T14:30:00Z",
      "result_id": "result-uuid-..."
    }
  ]
}
```

### Knowledge Base Stats

```bash
curl -H "X-API-Key: your-key" \
  https://api.krawl.sh/memory/stats
```

### Delete a Memory

```bash
curl -X DELETE -H "X-API-Key: your-key" \
  https://api.krawl.sh/memory/MEMORY_UUID
```

**Response:**
```json
{"deleted": true, "id": "MEMORY_UUID"}
```

## SSE Events

### `memory_recall`

Emitted at the start of research when prior memories are found:

```json
{
  "memories_count": 5,
  "memories": [
    {
      "content": "Ethereum merged to PoS in Sep 2022",
      "tags": ["ethereum", "proof-of-stake"]
    }
  ]
}
```

## Memory Entry Structure

Each memory entry contains:

| Field | Type | Description |
|---|---|---|
| `id` | string | UUID |
| `content` | string | The extracted fact/finding (1–2 sentences) |
| `source_query` | string | The original research query that produced this learning |
| `source_urls` | string[] | URLs of sources (up to 10) |
| `tags` | string[] | 1–5 topic tags (lowercase) |
| `created_at` | string | ISO timestamp |
| `result_id` | string\|null | ID of the research result that produced this memory |

## Scoping

Memories are scoped to the API key. Each API key has its own isolated knowledge base. When `KRAWL_DEV_KEY` is not set (no auth), all memories share a global namespace.

## Extraction Details

The memory extraction prompt instructs the model to:

- Extract 5–15 key facts per report
- Focus on data points, metrics, dates, names, relationships, and conclusions
- Make each learning self-contained
- Skip vague or opinion-based statements
- Add 1–3 topic tags per learning

The extraction uses the **first 8,000 characters** of the report to stay within Haiku's optimal context window.

---


# Export

Source: https://docs.krawl.sh/docs/export

Research results can be exported in seven formats via the `/results/\{result_id\}/export` endpoint.

## Formats

| Format | Content-Type | Use Case |
|---|---|---|
| `html` | `text/html` | Browser viewing, embedding, sharing |
| `pdf` | `application/pdf` | Download, print, archive |
| `markdown` | `text/markdown` | Raw content, further processing |
| `docx` | `application/vnd.openxmlformats-officedocument.wordprocessingml.document` | Editable Word document |
| `slides` | `text/html` | Self-contained reveal.js presentation deck |
| `run-record` | `application/json` | Shared run record and explicit unassessed claims |
| `evidence-pack` | `application/zip` | Unsigned evidence archive with offline verification |

## Usage

### HTML Export

```bash
curl -H "X-API-Key: your-key" \
  "https://api.krawl.sh/results/RESULT_UUID/export?format=html"
```

Supports a `theme` parameter for light or dark mode:

```bash
# Dark theme
curl -H "X-API-Key: your-key" \
  "https://api.krawl.sh/results/RESULT_UUID/export?format=html&theme=dark"
```

The HTML export is a self-contained page with:
- Full markdown-to-HTML rendering
- Styled headings, tables, lists, code blocks
- Source list with URLs and source types
- Research metadata (query, mode, duration)

### PDF Export

```bash
curl -H "X-API-Key: your-key" \
  "https://api.krawl.sh/results/RESULT_UUID/export?format=pdf" \
  -o report.pdf
```

The PDF includes:
- Title and metadata header
- Full report content with formatting
- Sources appendix with URLs and source types
- Generated via fpdf2 with HTML subset rendering
- Fallback to plain text if HTML rendering fails

### Markdown Export

```bash
curl -H "X-API-Key: your-key" \
  "https://api.krawl.sh/results/RESULT_UUID/export?format=markdown" \
  -o report.md
```

Returns the raw markdown report as generated by the synthesis model. This is the most portable format for further processing.

### DOCX Export

```bash
curl -H "X-API-Key: your-key" \
  "https://api.krawl.sh/results/RESULT_UUID/export?format=docx" \
  -o report.docx
```

Returns an editable Microsoft Word document with the title, metadata, formatted report body, and a sources appendix.

### Slides Export

```bash
curl -H "X-API-Key: your-key" \
  "https://api.krawl.sh/results/RESULT_UUID/export?format=slides" \
  -o report-slides.html
```

Returns a self-contained [reveal.js](https://revealjs.com) HTML deck that turns the report into a presentation. Open it in any browser — no build step required. Long sections are split across continuation slides automatically.

## Parameters

| Parameter | Type | Default | Options |
|---|---|---|---|
| `format` | string | `"html"` | `html`, `pdf`, `markdown`, `docx`, `slides`, `run-record`, `evidence-pack` |
| `theme` | string | `"light"` | `light`, `dark` (HTML only) |

## Filenames

Downloaded files use a sanitized version of the research query as the filename:

- `what-is-the-current-state-of-ai-agents.pdf`
- `what-is-the-current-state-of-ai-agents.md`
- `what-is-the-current-state-of-ai-agents.docx`
- `what-is-the-current-state-of-ai-agents-slides.html`

HTML and slides exports use `Content-Disposition: inline` (displayed in browser). PDF, Markdown, and DOCX use `Content-Disposition: attachment` (downloaded).

## Getting a Result ID

Result IDs are returned in the `result` SSE event during research, and can be retrieved from:

```bash
# List all results
curl -H "X-API-Key: your-key" https://api.krawl.sh/results

# Get a specific result
curl -H "X-API-Key: your-key" https://api.krawl.sh/results/RESULT_UUID
```

## Evidence time and access

Research sources carry optional `retrieved_at` (UTC), `published_date`,
`updated_date`, date provenance, `access_status`, `source_record_id` and
`temporal_status`. Missing fields on older reports mean unknown; never assign
export time as retrieval time. Access states are `inspected_text`,
`partial_excerpt`, `provider_summary`, `metadata_only`, `fetch_failed`, or
`unknown`. These describe available evidence, not claim support or truth.
A recent fetch does not make an old article current. Summary/metadata-only
results do not establish that the original text was read.

Optional `source_policy.as_of` sets a UTC evidence cutoff; dates after it are
quarantined in `quarantined_sources`, outside synthesis. Without it the cutoff
is research start. This is not a historical web archive; undated content
remains uncertain. See [evidence methodology](https://docs.krawl.sh/docs/trust#evidence-time-and-access).

## Evidence coverage

The optional `evidence_coverage` result field gives descriptive counts of unique
exact URLs discovered, explicitly fetched, retained, referenced in outgoing
model input, and cited. Stages overlap: a retained search snippet may never
have had its original page fetched. It also reports explicit failed/unavailable
fetches, unknown fetch/access states, retained-host concentration, missing
publication dates and their known range. A model-input reference may contain
only metadata or a short excerpt; a citation is not a support verdict.

These counts do not establish representative sampling or correct selection
bias. Hostnames do not prove independent origins; mirrors may share a publisher.
Old reports without this field have unknown coverage. No extra LLM call is made;
the measured instrumentation overhead is reported as `measurement_overhead_ms`,
not a guarantee of zero added latency. See [evidence methodology](https://docs.krawl.sh/docs/trust#evidence-coverage).

## Run record

Download `/results/{result_id}/export?format=run-record` with the owning API key.
The JSON bundle uses schema version 1 and includes the run, claim candidates,
source observations, a report snapshot, and the original Markdown. It is
compatible with the shared Claude Code run-record schemas. Every hosted claim
starts `unassessed`; this export does not judge support or reconstruct saved
page bytes. Empty events and ledger arrays mean those details were not retained,
not that the run was free. Legacy observation dates remain unknown.

MCP `export_result` accepts the same format and returns JSON text inline. The
read-only export does not add storage or make ZDR runs retrievable.

Schemas: [bundle](https://krawl.sh/schemas/run-record/bundle.schema.json).
From the repository, validate a downloaded bundle offline:

```bash
uv run --project apps/api --locked python scripts/validate_run_record.py report-run-record.json
```

## Evidence pack

Use `format=evidence-pack` to download an **unsigned** ZIP with a versioned
manifest, the run record, and instructions for offline verification. The
manifest binds exact stored bytes with SHA-256. It preserves source access
states and observation times; an export timestamp is never substituted for a
missing retrieval time. Hosted results have no retained raw page copies, so
those copies are explicitly **unverifiable**. An export does not fetch them.

```bash
curl -H "X-API-Key: your-key" \
  "https://api.krawl.sh/results/RESULT_UUID/export?format=evidence-pack" \
  -D export-headers.txt -o report.evidence.zip
```

The owning key is required for retained results. Wrong-owner and deleted
results return 404. The response is `application/zip`, with `Cache-Control:
private, no-store` and `X-Krawl-Evidence-SHA256`. Keep private packs private.
This operation creates no retained server-side archive or new source copy.

Current research SSE and MCP results include their server-allocated `id`,
`query`, `mode`, `created_at` and measured `duration_seconds`. The ID identifies
the logical result; it is not proof of persistence for a ZDR or failed-save run.
Cutoff and quarantined source observations are preserved in inline exports.

For a streamed result, including a ZDR result, send the full public result
object in `{"result": ...}` to `POST /exports/evidence-pack`. It must include
`id`, `query`, `mode`, `report` (or `markdown`), `sources`, `created_at` and
`duration_seconds`. The request body is limited to 1 MiB. The server projects
only public result fields and builds the archive in memory; it never stores the
query, content, source URL hashes, archive, or idempotency response. It sends no
webhook and makes no model or source-provider call. You may save the returned
archive on your own device. Supplying an object does not authenticate its
origin or assertions.

MCP `export_result` accepts `format: "evidence-pack"` and exactly one of
`result_id` or `result`. The ZIP is returned in `content_base64`, with `sha256`,
`manifest_sha256`, `size_bytes`, `unavailable_copies`, and `unsigned: true`.
Other export formats continue to use a retained `result_id`.

### Verify offline

From a checkout, install the independent core and run:

```bash
uv run --package krawl-core krawl-evidence verify report.evidence.zip
# Optionally pin a digest obtained through a trusted channel:
uv run --package krawl-core krawl-evidence verify report.evidence.zip --sha256 DIGEST
```

The verifier has no API credentials or network requirement and extracts no
files. Exit **0** means the declared bytes, internal references and available
cited passages match. Exit **2** means some source copies are unavailable; the
output identifies each source and report version. Exit **1** means invalid
bytes, metadata, references, passage offsets or archive structure. These codes
do not measure factual correctness, source independence or publisher identity.
An attacker who can replace the entire unsigned archive can replace its
hashes. A trusted external digest detects replacement of the whole archive.

For local Claude Code runs, the workbench's **Download evidence pack** includes
all report versions and already saved copies. The command-line equivalent is:

```bash
uv run --package krawl-core --extra engine krawl-engine export \
  --project /path/to/project --output report.evidence.zip
```

The CLI creates a new private file (0600); it refuses to overwrite a file.
Missing local captures remain explicitly unverifiable. The export preserves
existing retention choices and does not acquire new rights to redistribute a
source. Use only copies you are entitled to retain and share.

[Manifest schema](https://krawl.sh/schemas/run-record/evidence-pack.schema.json).
The format is version 1: SHA-256 over exact raw bytes and UTF-8 extracted text;
`json-utf8-sort-v1` encodes JSON with sorted keys, compact separators, unescaped
Unicode, no non-finite numbers, and one trailing newline. Passage offsets count
Unicode code points in the extracted text. The verifier rejects duplicate JSON
keys, traversal, symlinks, unexpected members, and oversized archives (64 MiB
content, 1 MiB manifest, 4,096 entries). It supports stored and deflated ZIP
members without extracting them.

Signed origin attestations are future work. They would require defined issuer
identity and claim semantics, trusted roots, signing-key custody and rotation,
revocation and compromise handling, and client verification policy. This
unsigned format provides none of those assurances.

---


# Scheduled Lookouts

Source: https://docs.krawl.sh/docs/lookouts

Lookouts are recurring research tasks that run on a schedule, detect changes between runs, and optionally notify you via webhook. Set a query, pick a schedule, and Krawl keeps watching.

## How Lookouts Work

1. **Create** a lookout with a research query, mode, and schedule
2. **Krawl runs the research** automatically on the schedule
3. **Results are compared** with the previous run to detect changes (LLM-powered diff)
4. **Webhooks fire** at each lifecycle stage: started, progress, completed, error

## Create a Lookout

```bash
curl -X POST https://api.krawl.sh/lookouts \
  -H "Content-Type: application/json" \
  -H "X-API-Key: your-key" \
  -d '{
    "query": "Latest developments in AI agents",
    "mode": "deep",
    "schedule": "daily",
    "webhook_url": "https://your-webhook.example.com/notify"
  }'
```

**Response:**
```json
{
  "id": "550e8400-...",
  "query": "Latest developments in AI agents",
  "mode": "deep",
  "schedule": "daily",
  "active": true,
  "last_run": null,
  "last_result_id": null,
  "created_at": "2025-01-15T10:00:00Z",
  "webhook_url": "https://your-webhook.example.com/notify"
}
```

## Schedules

| Schedule | Frequency |
|---|---|
| `hourly` | Every hour |
| `daily` | Once per day |
| `weekly` | Once per week |

## Limits

| Setting | Default | Env Variable |
|---|---|---|
| Max active lookouts per user | 10 | `LOOKOUT_MAX_PER_USER` |
| Minimum interval between runs | 60 minutes | `LOOKOUT_MIN_INTERVAL_MINUTES` |
| Manual trigger rate limit | 2/minute | — |

## List Lookouts

```bash
curl -H "X-API-Key: your-key" \
  "https://api.krawl.sh/lookouts?offset=0&limit=50"
```

## Get a Lookout

```bash
curl -H "X-API-Key: your-key" \
  https://api.krawl.sh/lookouts/LOOKOUT_UUID
```

## Delete a Lookout

```bash
curl -X DELETE -H "X-API-Key: your-key" \
  https://api.krawl.sh/lookouts/LOOKOUT_UUID
```

## Manual Trigger

Force an immediate run (respects the cooldown interval):

```bash
curl -X POST -H "X-API-Key: your-key" \
  https://api.krawl.sh/lookouts/LOOKOUT_UUID/run
```

Returns 429 if the cooldown hasn't elapsed since the last run.

## Change Detection

When a lookout runs, Krawl compares the new report with the previous one:

- **If identical** → `has_changes: false`, no diff
- **If different** → LLM-powered structured diff with:
  - `new_findings` — facts present in new report but not old
  - `removed_claims` — facts in old report that disappeared
  - `changed_data` — metrics/data that changed (with old and new values)
  - `summary` — natural language summary of changes

If the LLM diff fails, falls back to a simple line-level diff.

## Webhooks

When `webhook_url` is set, Krawl sends POST requests at each lifecycle stage.

### `started` Event

Fired when the lookout research begins.

```json
{
  "event": "started",
  "lookout_id": "...",
  "query": "Latest developments in AI agents",
  "timestamp": "2025-01-15T10:00:00Z"
}
```

### `progress` Event

Fired periodically during research with step count.

```json
{
  "event": "progress",
  "lookout_id": "...",
  "steps_completed": 12,
  "timestamp": "2025-01-15T10:01:00Z"
}
```

### `completed` Event

Fired when research finishes, includes the diff.

```json
{
  "event": "completed",
  "lookout_id": "...",
  "query": "Latest developments in AI agents",
  "result_id": "result-uuid",
  "has_changes": true,
  "diff": {
    "has_changes": true,
    "summary": "3 new findings about agent frameworks...",
    "new_findings": ["GPT-5 based agents announced", "..."],
    "removed_claims": [],
    "changed_data": []
  },
  "result_preview": "## AI Agents Report\n\nFirst 500 chars...",
  "steps_completed": 24,
  "timestamp": "2025-01-15T10:05:00Z"
}
```

### `error` Event

Fired if the research fails.

```json
{
  "event": "error",
  "lookout_id": "...",
  "query": "Latest developments in AI agents",
  "error": "Research pipeline error: ...",
  "timestamp": "2025-01-15T10:02:00Z"
}
```

### Webhook Security

- Webhook URLs are validated for format correctness
- Private/internal IPs are blocked (SSRF prevention)
- Webhooks fire asynchronously (non-blocking)

## Lookout Data Model

```json
{
  "id": "string (UUID)",
  "query": "string",
  "mode": "deep|quick|crypto|...",
  "schedule": "hourly|daily|weekly",
  "active": true,
  "last_run": "ISO timestamp or null",
  "last_result_id": "UUID or null",
  "created_at": "ISO timestamp",
  "webhook_url": "URL or null"
}
```

## Scheduler

Lookouts are managed by an internal scheduler that starts on application boot. The scheduler:

- Loads all active lookouts from the database
- Registers cron jobs for each schedule
- Respects the concurrency semaphore (max 3 concurrent research sessions)
- Updates `last_run` and `last_result_id` after each run

---


# Deployment

Source: https://docs.krawl.sh/docs/deployment

Krawl uses an AWS release path for the API, web app, and docs. Fly is retired.
The [current release status](https://github.com/ravidsrk/krawl/blob/main/docs/completion/RELEASE_STATUS.md)
records the evidence for each environment. Infrastructure creation and passing
CI alone do not establish a production release.

## Deployment layout

The repository is a monorepo with separate Docker images:

| App | Build context | Runtime |
| --- | --- | --- |
| API | Repository root, `apps/api/Dockerfile` | Python 3.12 / FastAPI on port 8080 |
| Web | `apps/web/` | React / Vite, served by nginx on port 8080 |
| Docs | `apps/docs/` | Next.js standalone on port 3000 |

[Terraform](https://github.com/ravidsrk/krawl/tree/main/infra/aws) defines a
separate ECS cluster per environment, immutable ECR repositories, an HTTPS
Application Load Balancer, scoped IAM roles and CloudWatch logs. Services start
with zero tasks; the release workflow activates them after infrastructure and
DNS are ready. API, web and docs each run one task. No autoscaler is configured.

Staging has an isolated VPC, encrypted private RDS PostgreSQL 16 with seven-day
backups, an AWS-managed database password and its own generated hash secret.
The API task reads the current database password for each new physical
connection. Secret values do not belong in GitHub variables, Terraform input
files or release artifacts.

## Prepare staging

Follow the repository's [AWS runbook](https://github.com/ravidsrk/krawl/blob/main/docs/operations/AWS_DEPLOYMENT.md)
for the reviewed account, region, cost estimate, issued ACM certificate, remote
state and exact Terraform plan. Only the staging hostnames should change during
staging setup:

- `https://staging.krawl.sh`
- `https://api.staging.krawl.sh`
- `https://docs.staging.krawl.sh`

Keep existing production DNS records and production data untouched. Configure
the GitHub environment `aws-staging`, restrict deployment branches to `main`,
and set these non-secret variables from the Terraform outputs:

| Variable | Purpose |
| --- | --- |
| `AWS_RELEASE_ROLE_ARN` | Environment-scoped GitHub OIDC role |
| `AWS_RELEASE_CONFIG` | Exact JSON release manifest with service, image and URL references |
| `AWS_RELEASE_ENABLED` | Set `true` after infrastructure, certificate and DNS readiness |

Provider invocation permissions default off. Supplying AWS hosting access does
not enable funded research or payment acceptance. Configure provider credentials
through Secrets Manager and record the approved evaluation budget separately.

## Release a reviewed commit

Run the **Deploy AWS** GitHub Actions workflow from `main`, select `staging`,
supply a full 40-character commit SHA and acknowledge the API maintenance window.
The exact SHA must have a successful push CI run on `main`. The workflow assumes
its environment role through OIDC; GitHub does not need long-lived AWS keys.

The workflow publishes missing images with immutable SHA tags, resolves their
ECR digests, registers task definitions, and updates API, docs and web in order.
Retries reuse existing image tags. The web image is built with the destination
API origin; a new origin requires a new reviewed commit. Docs use the configured
API origin for their MCP proxy.

**API replacement can interrupt in-flight work.** The single API task stops
before its replacement starts, with a bounded shutdown period. Schedule a quiet
maintenance window. Do not increase API task count until overlapping startup
recovery, session ownership and rate limits support it.

## Verify the deployed revision

A stable ECS service is only one check. The public endpoints must also agree
with the requested source revision:

- API `/health`: HTTP 200, `status=ok`, `db=ok`, and `X-Krawl-Revision` equal to the release SHA.
- Web `/release.json`: matching `revision` and the expected `api_url`.
- Docs `/release.json`: matching `revision`. The root redirects to `/docs`; the load balancer probes `/docs` for HTTP 200.

Both AWS environments require an API key. Anonymous or unknown-key research
requests must fail before execution. Separately check actual sign-in, key
issuance, account isolation, exports and erasure using synthetic accounts.
A synthetic report proves those storage and export paths; it does not prove
research quality, paid provider execution or billing reconciliation.

## Failure and rollback

The workflow saves an `aws-{environment}-release-{sha}` artifact containing
source revision, image digests, prior task definitions, task counts and rollback
outcomes. Task-registration failures record `preparation_failed` before services
change. Activation or health failures restore every touched service's prior task
and desired count, including zero-task bootstrap states.

Rollback preserves concurrent operator changes and reports unresolved outcomes
as `failed_requires_operator`. A failed release remains failed even if task
restoration succeeds. Confirm the prior public revision after restoration.
Task rollback does not undo database migrations or external effects.

CloudWatch logs retain 30 days under the infrastructure configuration. Review
ECS events, target health, API database readiness and the saved release artifact
when diagnosing a failure. Keep previous image digests available for recovery.

## Production cutover

Production requires verified existing database and unchanged hash-secret
references, verified network connectivity, and a continuity record. The current
production module creates neither a database nor a replacement hash secret.
Do not substitute staging data or rotate the hashing secret during migration;
that would orphan existing API keys and result ownership.

The `aws-production` environment also requires `AWS_PRODUCTION_DATA_VERIFIED`,
`AWS_STAGING_ACCEPTANCE_SHA` for the exact release commit, and a repository
`AWS_STAGING_ACCEPTANCE_URL`. Complete the binding release gates, backup and
restore checks, funded critical flows and owner decisions before production DNS
cutover. See the [release definition](https://github.com/ravidsrk/krawl/blob/main/docs/completion/DEFINITION.md).

The previous [Fly deployment guide](https://github.com/ravidsrk/krawl/blob/main/docs/operations/DEPLOYMENT.md)
is retained as historical material. Its credentials and commands are not the
AWS release procedure.

---


# Testing

Source: https://docs.krawl.sh/docs/testing

## Running Tests

```bash
# The API and its tests live in apps/api
cd apps/api

# Install dev dependencies
pip install -e ".[dev]"

# Run all tests
pytest

# Run with verbose output
pytest -v

# Run a specific test file
pytest tests/test_api.py

# Run tests matching a pattern
pytest -k "test_research"
```

## Test Files

| File | Coverage |
|---|---|
| `test_api.py` | API endpoint tests (health, research, search, results, export) |
| `test_api_new.py` | Additional API tests (templates, steering, sessions, compare) |
| `test_charts.py` | Data visualization extraction and rendering |
| `test_config.py` | Settings and environment variable loading |
| `test_conflict_detection.py` | Conflict detection between sources |
| `test_crypto_research.py` | Crypto research tools and modes |
| `test_db.py` | Database operations (results, sessions, lookouts, memory) |
| `test_export.py` | PDF, HTML, and Markdown export |
| `test_lookouts.py` | Scheduled research and change detection |
| `test_memory.py` | Knowledge base — memorize, recall, compare |
| `test_models.py` | Pydantic model validation |
| `test_prompts.py` | Prompt template generation |
| `test_research_engine.py` | Research engine — planning, agent loop, recursive search, synthesis |
| `test_scheduler.py` | Lookout scheduler lifecycle |
| `test_structured_output.py` | JSON Schema structured output extraction |
| `test_visualization.py` | Chart data extraction and spec generation |

### Client Tests

```
tests/test_clients/
├── test_browse.py        # URL content extraction
├── test_crypto.py        # CoinGecko, DeFi Llama clients
├── test_exa.py           # Exa search client
├── test_github.py        # GitHub search and code search
├── test_media.py         # YouTube search and transcription
├── test_x_search.py      # X/Twitter search via xAI
└── ...
```

### End-to-End Tests

```
web/e2e/
├── landing.spec.ts       # Landing page flows
├── research-input.spec.ts # Research input flows
└── streaming.spec.ts     # SSE streaming flows
```

Run from `web/`: `npx playwright install --with-deps chromium && npx playwright test`
(same as CI). The old Python `tests/e2e/` dir was removed — it was uncollectable
and never executed.

## Test Architecture

Tests use `pytest` with `pytest-asyncio` for async tests. Key patterns:

- **Mocked LLM calls** — all litellm calls are mocked to avoid API costs
- **Mocked HTTP clients** — external API calls (Exa, CoinGecko, etc.) are mocked
- **In-memory database** — database tests use a test PostgreSQL or mock the db module
- **FastAPI TestClient** — API tests use Starlette's TestClient for synchronous HTTP testing

## Running Specific Categories

```bash
# API tests only
pytest tests/test_api.py tests/test_api_new.py -v

# Crypto tests only
pytest tests/test_crypto_research.py tests/test_clients/test_crypto.py -v

# Research engine
pytest tests/test_research_engine.py -v

# Memory and knowledge base
pytest tests/test_memory.py -v

# Export formats
pytest tests/test_export.py -v
```

## Coverage

```bash
pytest --cov=app --cov-report=html
open htmlcov/index.html
```