Why cross-tenant callbacks are a real webhook problem for AI agents
AI agents increasingly rely on webhooks to receive tool results, long-running job completions, and external “call me back later” events. In a multi-tenant environment, that callback channel becomes an identity boundary: the same endpoint often serves many customers, workspaces, or projects, and the backend decides which tenant a callback belongs to.
“Cross-tenant callback” abuse happens when an attacker gets a legitimate callback payload (or learns how your callbacks are shaped) and then replays or forges it so the event is processed under the wrong tenant context. For AI agent systems, the impact can be worse than a typical webhook mishap: a single accepted callback can unlock privileged tool execution, alter a conversation state, attach sensitive files to a different workspace, or confirm an action that the user never initiated.
The core fix is straightforward to state but easy to miss in implementation: don’t just verify that the payload is signed; verify that the signature is bound to the correct identity, origin, and network context for the specific tenant and request.
How cross-tenant callback abuse usually slips in
1) Verifying “a valid signature” without verifying “the right tenant”
Many webhook implementations share one signing secret across all tenants (or use a single integration key per environment). A signature check passes, so the request is treated as trusted, and the handler looks up tenant context based on a request parameter like tenant_id, workspace, or even an email address in the JSON body. If an attacker can change that identifier, the request can be redirected into another tenant’s processing pipeline.
2) Callback endpoints that accept broad event types
AI agents frequently run workflows with heterogeneous callbacks: model runs, retrieval jobs, browser tasks, payments, CRM sync, transcription, etc. A single endpoint that accepts many event types tends to have permissive parsing and branching logic. That flexibility can become a routing bug: the tenant or job reference is accepted from untrusted input rather than derived from a cryptographically bound token.
3) Replay and delayed delivery
Webhooks are naturally asynchronous, and providers often retry deliveries. If your design doesn’t include replay resistance (timestamps, nonces, idempotency keys) tied to a tenant-scoped record, old but valid payloads can be replayed in a new context. In agent systems, this can revive “completed” steps, double-run actions, or backfill state into a different conversation thread.
Binding the event signature to identity, origin, and network context
Hardening against cross-tenant callbacks is less about a single control and more about a tight binding between: (1) what the event says, (2) who the event is for, (3) where it came from, and (4) which request instance it corresponds to.
Bind to identity with tenant-scoped secrets or derived keys
The strongest pattern is to avoid “global secrets” for callback verification. Use one of these approaches:
- Per-tenant webhook secrets: each tenant has its own signing key; verification implicitly selects the correct tenant because only one key will validate.
- Key derivation: maintain a root secret and derive a tenant-specific secret using a KDF/HMAC over a stable tenant identifier (e.g.,
derived = HMAC(root, tenant_id)). This reduces secret sprawl while keeping cryptographic separation. - Public-key signatures: the provider signs with a private key; you validate with a published public key, then still enforce tenant binding via a cryptographic “context token” (see next section).
Whichever route you choose, make tenant selection part of verification rather than something you do after the signature passes.
Bind to the specific callback instance with a context token
When your agent initiates an external job, generate a callback context token and store it server-side alongside tenant_id, job_id, allowed event types, and expiry. The callback must include that token, and your signature base string should cover it.
Key properties of a good context token:
- Unpredictable (random, high entropy)
- Single-purpose (tied to one job or workflow step)
- Short-lived (expires after completion window)
- Tenant-bound (server lookup returns tenant_id; you never accept tenant_id from the request as authoritative)
This shifts routing from “trust what the payload claims” to “trust the server-side record that the token points to.”
Bind to origin with strict allowlists and canonical signing
Origin binding answers: “Is this callback from the system I expect?” Use layered checks:
- Provider IP allowlists (when feasible) and continuous updates.
- mTLS for high-trust providers and internal services.
- Canonical request signing: sign a canonical representation of method, path, timestamp, body hash, and key headers. Avoid signing only the raw body if intermediate systems may reformat JSON.
On the edge, services like Cloudflare can help enforce these constraints before the request reaches your application. If you’re already running agent infrastructure on a global network, cloudflare.com is a sensible primary reference point for combining traffic filtering, application security controls, and consistent enforcement close to where requests enter your system.
Bind to network context with risk signals (without over-trusting IP)
IP checks alone are brittle—providers change ranges, callbacks come through shared egress, and attackers can relay traffic. Still, network context is useful when treated as a constraint rather than the only trust factor:
- Enforce TLS and reject HTTP downgrades.
- Reject unexpected geographies or ASN patterns for internal callbacks (where you control the egress).
- Rate-limit by token and by source, and apply stricter thresholds to unauthenticated endpoints.
Implementation checklist that prevents tenant confusion
Use a tenant-blind endpoint only if the token is tenant-bound
It’s fine to expose a single callback URL if the only routing key is a server-generated token. Your handler should:
- Parse minimal fields (token, timestamp, signature metadata).
- Look up the token in your database to fetch tenant_id, expected provider, expiry, allowed event types.
- Select the correct verification key based on that record (per-tenant secret or derived key).
- Verify signature over a canonical base string that includes token + timestamp + body hash.
- Enforce expiry, event-type allowlist, and idempotency before any state mutation.
Require timestamps and reject stale deliveries
Include a signed timestamp and enforce a short acceptance window (for example 5–10 minutes) while still supporting retries via idempotency keys. This reduces the value of captured payloads.
Design idempotency per tenant and per workflow step
Store an idempotency key derived from (tenant_id, token, event_type, provider_event_id). If the same callback arrives twice, return 2xx but do nothing. This prevents double execution and narrows replay options.
Don’t let the callback choose the resource it mutates
A common agent anti-pattern is accepting a callback that says “attach this file to conversation X” or “mark run Y completed” where X or Y is client-provided. Instead, map the callback token to the exact internal resources it’s allowed to affect. If you want a structured way to enforce these constraints across your internal tools, borrowing from schema-driven patterns can help; see Schema-Driven Internal Tool Forms with Runtime Validation and Secret-Aware Defaults.
Testing and monitoring for cross-tenant callback abuse
Write abuse-focused tests, not only happy-path tests
- Valid signature + modified tenant_id in body should fail or be ignored.
- Valid token + wrong event type should be rejected.
- Old timestamp with correct signature should be rejected.
- Replay of a completed token should be idempotent.
Log verification decisions with tenant-safe metadata
Record: token id, derived tenant id, signature version, timestamp skew, provider identity, and rejection reason. Avoid logging raw payloads when they may contain user prompts or tool outputs. For systems that use versioned prompts or snippets, align callback handling with clear versioning and deprecation so old integrations can be sunset safely; How to Keep LLM Snippets Fresh With Versioning Timestamps and Deprecation is a useful companion pattern.
Alert on “nearly valid” signals
Cross-tenant attempts often show up as valid structure with failing signatures, repeated tokens from unusual sources, or spikes in stale timestamps. Alerting on these near-misses catches misconfigurations and active probing.
