Where we are
In the Distributed Task Queue walkthrough we built workers that pull jobs from a queue we own. Today the other end of the wire belongs to someone else. A webhook system sends your events to thousands of servers you don't control, and some of them will be slow, broken or gone.
By the end of this lesson you'll have a working sender and receiver that pass eight checks: nothing lost, nothing processed twice, one bad customer can't slow the rest, and forged or replayed requests bounce.
The problem in one picture
When a payment succeeds, Stripe POSTs an invoice.paid event to your server. That sounds like one HTTP call. In practice it is a promise: "this message will arrive, even if your server is down for a day."
Each big platform has written down how far that promise goes:
Stripe retries for up to three days with exponential backoff in live mode, and says plainly that it doesn't guarantee events arrive in order (Stripe docs).
Shopify gives your endpoint five seconds to answer, retries up to eight times over four hours, and removes the subscription if failures continue (Shopify docs).
GitHub waits 10 seconds and does not retry automatically at all. You redeliver failures yourself (GitHub docs).
Three companies, three very different answers. The interviewer wants to hear you choose one on purpose.
Core idea: four promises
A webhook system makes four promises. Every design decision traces back to one of them.
Durable. If the event happened, it will be sent. No event is lost because a process crashed between "save the order" and "send the webhook."
Isolated. One customer's slow server never delays anyone else.
Verifiable. The receiver can prove the request came from you, wasn't changed, and isn't an old one replayed.
Recoverable. When delivery finally gives up, the event is parked, not deleted, and can be replayed later.
Notice what is missing: "exactly once." Over a network you can't promise that. You deliver at least once and give the receiver what it needs to ignore duplicates. Stripe's docs say the same: endpoints "might occasionally receive the same event more than once."
