Improve operator visibility for terminally failed deliveries

[quangshuynh]

Problem

Hymical Forms durably records webhook delivery obligations and retries failed attempts with bounded backoff. Once the retry budget is exhausted, however, a delivery can remain in its terminal failed state until an operator explicitly discovers it through the existing inspection surfaces.

This preserves the delivery record, but creates an operational visibility gap: durable failure is not the same as making that failure apparent to the operator.

For a service whose core responsibility is reliable webhook delivery, terminal failure should produce a clear, bounded, operator-visible signal without changing the existing at-least-once delivery model.

Goal

Determine and implement the smallest operational mechanism that makes newly terminal delivery failures clearly visible to an operator.

The solution should build on the existing delivery state and attempt history rather than introduce a separate delivery system.

Acceptance criteria

• A delivery that exhausts its retry budget produces an explicit operator-visible signal.
• The signal contains enough stable context to identify the delivery and endpoint without exposing submitted form contents unnecessarily.
• The signal is emitted only when the delivery transitions into terminal failure, rather than on every ordinary retry failure.
• Existing retry, replay, lease, and at-least-once semantics remain unchanged.
• Automated tests prove the terminal transition produces the signal and non-terminal failures do not.
• Relevant operations/limitations documentation explains how operators discover and respond to terminal failures.

Non-goals

• Prometheus or another metrics stack.
• A monitoring dashboard.
• A general notification framework.
• Email, Slack, or other third-party alert integrations.
• Automatic replay of terminally failed deliveries.
• Exactly-once delivery.
• Changes to retry policy solely for this issue.

Validation

The implementation should demonstrate the complete transition from a retryable delivery failure to terminal failure and verify that the resulting operational signal is produced exactly at that boundary.

If inspection shows that the existing operational surfaces already provide a sufficiently strong terminal-failure signal, close this issue without implementation and document the evidence.