APIStatusGitHub

Webhooks

Outbound webhook events for async research and scheduled lookouts, the signed payload contract, signature verification, and replay protection

Krawl delivers outbound webhooks so agents don't have to hold an SSE connection or poll. Two features emit them:

  • Async research (POST /research/async) — pass a webhook_url and Krawl notifies you when the background task finishes (or fails).
  • Scheduled lookouts (Lookouts) — recurring research jobs fire webhooks at each lifecycle stage: started, progress, completed, error.

Webhooks are delivered as an HTTP POST with a JSON body to the webhook_url you registered on the resource. Delivery is fire-and-forget: Krawl does not currently retry failed deliveries, so your endpoint should respond quickly (within ~10s) and 2xx. Treat webhooks as an optimization on top of the authoritative state you can always fetch via GET /tasks/{id} (async) or GET /lookouts/{id} + GET /results/{id} (lookouts).

Outbound vs. inbound. This page documents Krawl's own outbound webhooks. The separate Stripe billing webhook (POST /billing/webhook) is inbound — it receives events from Stripe and verifies Stripe's own signature. That is unrelated to the signing scheme below.

Event payloads

Every webhook body is a JSON object with an event field and a timestamp (RFC 3339 / ISO 8601, UTC). The other fields depend on the source.

Async research task events

Registered via webhook_url on POST /research/async.

completed — the background task finished and a result was saved:

{
  "event": "completed",
  "task_id": "550e8400-e29b-41d4-a716-446655440000",
  "result_id": "9f1c2b34-...",
  "query": "Latest developments in AI agents",
  "timestamp": "2025-01-15T10:05:32.184Z"
}

error — the task failed (or the server had no capacity to start it):

{
  "event": "error",
  "task_id": "550e8400-e29b-41d4-a716-446655440000",
  "error": "server_busy",
  "timestamp": "2025-01-15T10:05:32.184Z"
}

Fetch the full report with GET /results/{result_id} once you receive a completed event, or re-check status with GET /tasks/{task_id}.

Lookout events

Registered via webhook_url on POST /lookouts. A single run emits these in order: started → progress (repeating) → completed (or error).

started:

{
  "event": "started",
  "lookout_id": "7b3e...",
  "query": "Latest developments in AI agents",
  "timestamp": "2025-01-15T10:00:00.000Z"
}

progress (emitted roughly every 5 research steps):

{
  "event": "progress",
  "lookout_id": "7b3e...",
  "query": "Latest developments in AI agents",
  "steps_completed": 5,
  "timestamp": "2025-01-15T10:01:12.000Z"
}

completed (includes the change diff vs. the previous run):

{
  "event": "completed",
  "lookout_id": "7b3e...",
  "query": "Latest developments in AI agents",
  "result_id": "9f1c2b34-...",
  "has_changes": true,
  "diff": {
    "has_changes": true,
    "summary": "Two new model releases since the last run.",
    "new_findings": ["..."],
    "removed_claims": ["..."],
    "changed_data": [{ "field": "...", "old": "...", "new": "..." }]
  },
  "result_preview": "First 500 characters of the report markdown...",
  "timestamp": "2025-01-15T10:05:00.000Z"
}

error:

{
  "event": "error",
  "lookout_id": "7b3e...",
  "query": "Latest developments in AI agents",
  "error": "research engine timeout",
  "steps_completed": 12,
  "timestamp": "2025-01-15T10:05:00.000Z"
}

Signature verification

Rollout status. Signed delivery is the documented contract for outbound webhooks. When signing is enabled for your endpoint, every delivery carries an X-Krawl-Signature header computed as below. Endpoints registered before signing was enabled may receive unsigned deliveries during the transition — verify the signature when the header is present, and plan to require it.

Krawl signs each webhook using HMAC-SHA256 over the raw request body with a per-endpoint signing secret (whsec_...). The signature is sent in the X-Krawl-Signature header using a Stripe-style scheme that also carries the signed timestamp:

X-Krawl-Signature: t=1736935500,v1=5257a869e7 ... 8e1f9c
  • t — Unix timestamp (seconds) when the signature was generated.
  • v1 — lowercase hex HMAC-SHA256 of the string "{t}.{raw_body}", keyed with your endpoint's signing secret.

The signed payload is the timestamp, a literal ., and the exact raw bytes of the request body:

signed_payload = "{t}" + "." + raw_request_body
expected_v1    = hex( HMAC_SHA256(signing_secret, signed_payload) )

To verify: recompute expected_v1 and compare it to the v1 value using a constant-time comparison. Reject the request if they differ.

Node.js

import crypto from "node:crypto";

// rawBody MUST be the exact bytes received (do not re-serialize parsed JSON).
function verifyKrawlWebhook(rawBody, signatureHeader, signingSecret, toleranceSec = 300) {
  const parts = Object.fromEntries(
    signatureHeader.split(",").map((kv) => kv.split("=")),
  );
  const t = Number(parts.t);
  const v1 = parts.v1;
  if (!t || !v1) throw new Error("malformed signature header");

  // Replay protection: reject stale timestamps.
  const ageSec = Math.abs(Date.now() / 1000 - t);
  if (ageSec > toleranceSec) throw new Error("timestamp outside tolerance");

  const expected = crypto
    .createHmac("sha256", signingSecret)
    .update(`${t}.${rawBody}`, "utf8")
    .digest("hex");

  const ok = crypto.timingSafeEqual(Buffer.from(expected), Buffer.from(v1));
  if (!ok) throw new Error("signature mismatch");
  return true;
}

// Express example — capture the raw body:
// app.use("/webhook", express.raw({ type: "application/json" }));
app.post("/webhook", (req, res) => {
  try {
    verifyKrawlWebhook(req.body.toString("utf8"), req.get("X-Krawl-Signature"), process.env.KRAWL_WEBHOOK_SECRET);
  } catch (e) {
    return res.status(400).send(`invalid signature: ${e.message}`);
  }
  const event = JSON.parse(req.body.toString("utf8"));
  // ... handle event.event ("completed", "error", ...)
  res.sendStatus(200);
});

Python

import hmac
import hashlib
import time


def verify_krawl_webhook(raw_body: bytes, signature_header: str, signing_secret: str, tolerance_sec: int = 300) -> bool:
    parts = dict(kv.split("=", 1) for kv in signature_header.split(","))
    t = int(parts["t"])
    v1 = parts["v1"]

    # Replay protection: reject stale timestamps.
    if abs(time.time() - t) > tolerance_sec:
        raise ValueError("timestamp outside tolerance")

    signed_payload = f"{t}.".encode() + raw_body
    expected = hmac.new(signing_secret.encode(), signed_payload, hashlib.sha256).hexdigest()

    if not hmac.compare_digest(expected, v1):
        raise ValueError("signature mismatch")
    return True


# FastAPI example:
# @app.post("/webhook")
# async def webhook(request: Request):
#     raw = await request.body()
#     verify_krawl_webhook(raw, request.headers["X-Krawl-Signature"], os.environ["KRAWL_WEBHOOK_SECRET"])
#     event = json.loads(raw)
#     ...

Verifying manually with curl / shell

You can reproduce the v1 value from a captured body and timestamp:

# t and raw body must match exactly what was delivered.
T=1736935500
BODY='{"event":"completed","task_id":"550e8400-...","result_id":"9f1c...","query":"...","timestamp":"2025-01-15T10:05:32.184Z"}'

printf '%s.%s' "$T" "$BODY" \
  | openssl dgst -sha256 -hmac "$KRAWL_WEBHOOK_SECRET" -hex \
  | sed 's/^.* //'
# Compare the output to the v1=... value in the X-Krawl-Signature header.

Replay protection

The signed timestamp (t) defends against replay attacks:

  1. Enforce a tolerance window (the examples use 300s / 5 minutes). Reject any delivery whose t is too far from your server clock — a captured-and-replayed request will eventually fall outside the window.
  2. Because t is part of the signed payload, an attacker cannot rewrite the timestamp without invalidating v1.
  3. For exactly-once processing, also deduplicate on the event's natural key (task_id / lookout_id + event), since retried or duplicated deliveries can occur.

Best practices

  • Always read the raw body before parsing JSON — re-serializing changes bytes and breaks the HMAC.
  • Respond fast (2xx) and process async. Krawl uses a ~10s delivery timeout and does not retry; do heavy work after acknowledging.
  • Fall back to polling. Webhooks are best-effort. The source of truth is GET /tasks/{id} (async) and GET /lookouts/{id} + GET /results/{id} (lookouts). See Rate Limits & Retries for backoff guidance.
  • Use HTTPS endpoints. Krawl validates webhook URLs against SSRF rules and will refuse to deliver to private/internal addresses.

Batch & Bulk Operations

Submit many research jobs in one call with POST /research/batch, or fan out parallel async tasks; plus the multi-result compare primitive

Rate Limits & Retries

Rate-limit response headers, the 429 error envelope, and an exponential-backoff retry playbook for autonomous agents

On this page

Event payloadsAsync research task eventsLookout eventsSignature verificationNode.jsPythonVerifying manually with curl / shellReplay protectionBest practices