TICKET HX-1212

Webhook subscription not delivering events

Status: Resolved Category: API Priority: P2 Customer: Lila Okafor (Pinetree Studios) Agent: Sandra Liu
Lila Okafor Customer 2024-03-25 16:30 UTC

I subscribed our endpoint to run.failed events 4 days ago and we haven't received a single delivery, even though I can see failed runs in the dashboard. Endpoint URL is https://api.pinetree.studio/helix-webhook. What's broken?

Sandra Liu Helix Support 2024-03-25 16:51 UTC

Hi Lila — looking now. I see your subscription. I also see we attempted 47 deliveries to that URL in the last 4 days. Every one returned a 403 Forbidden response from your endpoint, so they're being marked as failed deliveries.

You can confirm this yourself in Settings → Webhooks → {your-subscription} → Delivery Logs. After 3 retries we mark each event permanently failed.

The 403 suggests your endpoint is rejecting our requests — maybe a signature check failing, or an auth middleware blocking unknown clients. Can you check your access logs for what's returning 403?

Lila Okafor Customer 2024-03-25 17:09 UTC

You're right, our middleware is rejecting unsigned requests. We have signature verification on, expecting an HMAC in a custom header. What's the header name and signing algorithm Helix uses?

Sandra Liu Helix Support 2024-03-25 17:14 UTC

The header is X-Helix-Signature and the signing algorithm is HMAC-SHA256 over the raw request body, using the secret from your subscription config. Make sure you're verifying against the raw bytes, not the parsed JSON — re-serialization will produce different bytes and break signature verification.

Also: we don't backfill failed deliveries automatically. Once your endpoint is fixed, only new events going forward will arrive. If you need the events from the last 4 days, let me know and I can help you replay them.

Lila Okafor Customer 2024-03-25 18:02 UTC

Got signature verification working. Test event came through. Please replay the missed events from the last 4 days, that would help.

Sandra Liu Helix Support 2024-03-25 18:21 UTC

Replay queued — 47 events from the last 4 days are being redelivered. They'll trickle in over the next 30 minutes to avoid spiking your endpoint. Closing this out.

RESOLUTION

Customer's webhook endpoint was returning 403 because their middleware required a signed request and signature verification wasn't yet implemented on their side. Helix uses X-Helix-Signature header with HMAC-SHA256 over the raw body. Customer implemented verification correctly, agent replayed 4 days of missed deliveries.