APIStatusGitHub

Batch & Bulk Operations

Submit many research jobs in one call with POST /research/batch, or fan out parallel async tasks; plus the multi-result compare primitive

Krawl supports batch/bulk research three ways:

  1. POST /research/batch — submit up to 20 queries in a single request. Each is queued as an independent background task and you get one task handle per query back. This is the formal batch endpoint, defined in /openapi.json.
  2. Parallel async tasks — fan out many POST /research/async jobs and poll GET /tasks manually (the lower-level equivalent of the batch endpoint).
  3. POST /research/compare — a multi-result primitive that diffs two prior results in a single call (useful for change detection over a batch of saved results).

Batch endpoint — POST /research/batch

Submit an array of queries and receive a task handle for each. The server queues them concurrently (subject to a per-key concurrency cap) and never holds open SSE connections.

curl -X POST https://api.krawl.sh/research/batch \
  -H "X-API-Key: krawl_live_..." -H "Content-Type: application/json" \
  -d '{
    "queries": [
      {"query": "State of AI agents 2025", "mode": "deep"},
      {"query": "DeFi TVL trends Q1", "mode": "quick"},
      {"query": "Latest Llama benchmarks", "mode": "deep", "webhook_url": "https://example.com/hook"}
    ]
  }'

Response (202 Accepted):

{
  "accepted": 3,
  "tasks": [
    {"id": "550e8400-...", "query": "State of AI agents 2025", "mode": "deep", "status": "queued"},
    {"id": "661f9511-...", "query": "DeFi TVL trends Q1", "mode": "quick", "status": "queued"},
    {"id": "772fa622-...", "query": "Latest Llama benchmarks", "mode": "deep", "status": "queued"}
  ]
}
FieldTypeNotes
queriesarray (1–20)Each item: query (required), mode (deep/quick/crypto), optional webhook_url.
acceptedintNumber of jobs queued.
tasks[]arrayOne task handle per query — poll GET /tasks/{id}.

Then poll each task (see Pattern A) or use per-item webhook_url callbacks. If the batch would exceed your active-task cap, the whole request is rejected with 429 + Retry-After rather than partially queued — retry with backoff.

Pattern A — parallel async research

Submit each query as its own background task, then poll for completion. The server runs them concurrently (subject to a per-key concurrency cap) and you never hold open SSE connections.

1. Submit jobs

curl -X POST https://api.krawl.sh/research/async \
  -H "X-API-Key: krawl_live_..." -H "Content-Type: application/json" \
  -d '{"query": "State of AI agents 2025", "mode": "deep"}'
{
  "id": "550e8400-...",
  "query": "State of AI agents 2025",
  "mode": "deep",
  "status": "queued",
  "result_id": null,
  "created_at": "2025-01-15T10:00:00Z"
}

Repeat for each query in your batch, collecting the returned task ids.

Concurrency cap. Each API key can only have a bounded number of queued+running tasks at once. When you exceed it, submission returns 429 with a message like Too many active async tasks (N/MAX). Throttle submissions and retry with backoff, or wait for in-flight tasks to finish.

2. Poll for completion

Poll the collection (newest first) or individual tasks:

# All your tasks
curl -H "X-API-Key: krawl_live_..." "https://api.krawl.sh/tasks?limit=50&offset=0"

# A single task
curl -H "X-API-Key: krawl_live_..." "https://api.krawl.sh/tasks/550e8400-..."

Each task has a status of queued, running, done, or error. When done, fetch the report via GET /results/{result_id}:

{
  "id": "550e8400-...",
  "query": "State of AI agents 2025",
  "mode": "deep",
  "status": "done",
  "result_id": "9f1c2b34-...",
  "finished_at": "2025-01-15T10:06:12Z"
}

3. (Optional) get notified instead of polling

Pass webhook_url on submission to receive a completed/error callback per task instead of polling. See Webhooks.

Batch driver (pseudocode)

import time, httpx

queries = ["q1", "q2", "q3", "..."]
client = httpx.Client(base_url="https://api.krawl.sh", headers={"X-API-Key": "krawl_live_..."})

# Submit (respecting the concurrency cap: retry 429s with backoff)
task_ids = []
for q in queries:
    r = client.post("/research/async", json={"query": q, "mode": "deep"})
    r.raise_for_status()
    task_ids.append(r.json()["id"])

# Poll until all terminal
pending = set(task_ids)
while pending:
    for tid in list(pending):
        t = client.get(f"/tasks/{tid}").json()
        if t["status"] in ("done", "error"):
            pending.discard(tid)
            if t["status"] == "done":
                report = client.get(f"/results/{t['result_id']}").json()
                # ... use report
    time.sleep(5)  # poll interval; honor rate-limit headers

Use a polling interval (e.g. 5s) and watch X-RateLimit-Remaining so polling a large batch doesn't trip the rate limiter. See Rate Limits & Retries.

Pattern B — POST /research/compare

compare is the multi-result batch primitive for change detection: it takes results you already have and returns a structured diff in one call.

Request (CompareRequest):

FieldTypeDescription
querystringrequired — query to find/compare results for (1–2000 chars)
result_idsstring[]optional, up to 10 — specific results to compare

Behavior depends on how many result_ids you pass:

  • 2+ ids → compares the first two of those results directly.
  • 1 id → compares it against the most similar previous result for its query.
  • 0 ids → finds the two most recent results matching query and compares them.
curl -X POST https://api.krawl.sh/research/compare \
  -H "X-API-Key: krawl_live_..." -H "Content-Type: application/json" \
  -d '{"query": "AI agents", "result_ids": ["9f1c...", "7b3e..."]}'

Response:

{
  "old_result_id": "7b3e...",
  "new_result_id": "9f1c...",
  "old_query": "AI agents",
  "new_query": "AI agents",
  "old_created_at": "2025-01-08T10:00:00Z",
  "new_created_at": "2025-01-15T10:00:00Z",
  "changes": [{ "field": "...", "old": "...", "new": "..." }],
  "new_findings": ["..."],
  "removed_claims": ["..."],
  "summary": "Natural-language summary of what changed.",
  "has_changes": true
}

If fewer than two comparable results exist, compare returns 404 with a message telling you how many it found — submit more research first.

Choosing a pattern

GoalUse
Run many independent queries at oncePattern A (parallel /research/async + /tasks)
Detect what changed between two reportsPattern B (/research/compare)
Recurring watch + auto-diff on a scheduleLookouts

Trust

How Krawl handles citation, verification, failure modes, refunds, and audit — the methodology page an autonomous agent's owner should read before wiring Krawl into a production pipeline.

Webhooks

Outbound webhook events for async research and scheduled lookouts, the signed payload contract, signature verification, and replay protection

On this page

Batch endpoint — POST /research/batchPattern A — parallel async research1. Submit jobs2. Poll for completion3. (Optional) get notified instead of pollingBatch driver (pseudocode)Pattern B — POST /research/compareChoosing a pattern