Rate Limits & Retries

Krawl rate-limits requests per plan (and per IP for anonymous/demo traffic). This page documents the response headers that advertise your current limit state, the structured error envelope returned on failure, and a concrete retry/backoff playbook agents should implement.

Plan limitsCopy Anchor Link

Anonymous/unauthenticated traffic defaults to 5 requests/minute per IP. Some endpoints set their own tighter limits (e.g. manual lookout triggers at 2/min).

Separately, async fan-out is bounded: POST /research/async rejects new submissions with 429 once you have too many concurrent queued+running tasks.

Anonymous playground (POST /playground)Copy Anchor Link

The homepage demo at krawl.sh runs against POST /playground, a no-key, no-signup surface. Because it is unauthenticated it gets its own throttle layer on top of the global anonymous limit:

The session-priority trade-off. On a shared egress IP (corporate NAT, café, co-working space) a strict IP-only cap penalises the first visitor for traffic their colleague behind the same NAT has already used. The priority slot avoids that: each fresh browser cookie (krawl_pg_session, HttpOnly, lax, 1h TTL, scoped to /playground) gets 3 priority runs that bypass the per-IP 15/hour cap. After those 3 runs are spent, subsequent runs from the same cookie fall back to the IP cap.

Two things to know:

• Clearing cookies does not grant new slots beyond the IP cap. A freshly-minted cookie still consults the IP cap on its first run — the priority slot only protects returning visitors with priority left, not anonymous cookie-rotators. This is what makes the per-IP 15/hour a hard ceiling.
• The counter is process-local. Every Fly auto-suspend or machine restart resets the playground's per-IP and per-session windows. This is acceptable for a marketing surface (worst case is one extra request per restart) and keeps the playground from depending on Postgres.

If you need higher limits, get an API key or pay per-request via x402; the playground is an activation surface, not a free API tier.

Rate-limit headersCopy Anchor Link

Responses carry standard rate-limit headers so agents can self-throttle before hitting a 429:

Read X-RateLimit-Remaining on every response. When it approaches 0, pace yourself until X-RateLimit-Reset rather than firing requests into a 429.

The 429 responseCopy Anchor Link

When you exceed the limit, Krawl returns HTTP 429 with the standard error envelope and a Retry-After header:

HTTP/1.1 429 Too Many Requests
Retry-After: 60
X-RateLimit-Limit: 10
X-RateLimit-Remaining: 0
X-RateLimit-Reset: 1736935560
Content-Type: application/json

{
  "error_code": "RATE_LIMITED",
  "message": "Rate limit exceeded",
  "detail": "Rate limit exceeded",
  "retry_after": 60
}

Error envelopeCopy Anchor Link

All errors share one JSON shape so agents can parse failures uniformly and decide whether a retry is warranted:

{
  "error_code": "RATE_LIMITED",
  "message": "human-readable message",
  "detail": "...",
  "retry_after": 60
}

Error codes & retry eligibilityCopy Anchor Link

Rule of thumb: retry on 429 and 5xx. Do not retry 4xx other than 429 — those indicate a client-side problem the same request won't fix. A 402 means payment/quota is required (see Payments & x402).

Retry playbook (exponential backoff + jitter)Copy Anchor Link

Agents should implement bounded exponential backoff with full jitter, and honor Retry-After/retry_after when present (it overrides your computed delay).

import random
import time
import httpx

RETRYABLE_STATUS = {429, 500, 502, 503, 504}


def request_with_retry(client, method, url, *, max_retries=5, base=0.5, cap=30.0, **kwargs):
    for attempt in range(max_retries + 1):
        resp = client.request(method, url, **kwargs)
        if resp.status_code not in RETRYABLE_STATUS:
            return resp  # success or non-retryable error
        if attempt == max_retries:
            return resp  # give up, let caller handle

        # Prefer the server's hint when present.
        retry_after = resp.headers.get("Retry-After")
        if retry_after is not None:
            delay = float(retry_after)
        else:
            # Exponential backoff with full jitter.
            delay = random.uniform(0, min(cap, base * (2 ** attempt)))
        time.sleep(delay)
    return resp


with httpx.Client(base_url="https://api.krawl.sh", headers={"X-API-Key": "krawl_live_..."}) as c:
    r = request_with_retry(c, "POST", "/research", json={"query": "What is EIP-4844?", "mode": "quick"})

const RETRYABLE = new Set([429, 500, 502, 503, 504]);

async function requestWithRetry(url, init, { maxRetries = 5, base = 500, cap = 30000 } = {}) {
  for (let attempt = 0; ; attempt++) {
    const resp = await fetch(url, init);
    if (!RETRYABLE.has(resp.status) || attempt === maxRetries) return resp;

    const retryAfter = resp.headers.get("Retry-After");
    const delay = retryAfter
      ? Number(retryAfter) * 1000
      : Math.random() * Math.min(cap, base * 2 ** attempt); // full jitter
    await new Promise((r) => setTimeout(r, delay));
  }
}

GuidelinesCopy Anchor Link

• Cap retries (e.g. 5) and cap the delay (e.g. 30s) so a stuck agent doesn't loop forever.
• Always add jitter — synchronized retries from a fleet of agents cause thundering-herd spikes that re-trigger 429s.
• Honor the server hint. If Retry-After or retry_after is set, wait at least that long.
• Self-throttle proactively using X-RateLimit-Remaining / X-RateLimit-Reset to avoid 429s in the first place.
• For long-running work, prefer POST /research/async and poll GET /tasks/{id} instead of holding a connection — it spreads load and avoids retry storms. See Status & Uptime for the full recovery playbook.