Skip to main content

​
Context

Every queue is at-least-once. A worker will process the same job twice — after a timeout, a deploy, a provider retry — and in a system that awards points, that is money. Idempotency is not a quality attribute here; it is the correctness condition, which is why DEC-C6 made processed_jobs mandatory for every consumer rather than a per-worker choice. The second half is DEC-C5’s evidence guarantee: every failure and every exhausted retry is persisted, not merely logged. A log line rotates away; a dead_letters row is something an operator can find, understand and replay. Successes at volume are metrics, not rows — the asymmetry is deliberate.

​
Scope (normative)

  • core.processed_jobs: the dedupe fence, written inside the work transaction.
  • core.dead_letters: failures with payload, attempts, last error and review state.
  • The consumer harness: idempotency gate, retry with backoff and jitter, dead-letter writer.
  • TTL cleanup for processed_jobs.
  • BetterStack alerting on new dead letters.
  • Replay from Softcrum Ops, republishing with the same job_id.

​
Non-scope (normative)

  • QueuePort itself and its adapters — TS-002.
  • Notification rails — those are messaging, though they use this harness.
  • A UI for replay — frontend/ops.

​
Behaviour (normative)

  1. A consumer checks and inserts processed_jobs inside the same transaction as its work. Outside it, a crash between the two leaves the job marked done with nothing done.
  2. A duplicate job_id acks and skips. Silently, with a metric — a duplicate is normal operation, not an error.
  3. Retry policy: 5 attempts, exponential backoff with jitter, base 30 s, cap 1 h. Overriding it per queue requires a rationale in that module’s spec.
  4. Exhausted retries and hard failures write a dead_letters row with the full payload, attempt count, last error and correlation_id, and alert BetterStack.
  5. Poison messages skip retries entirely. A schema-invalid payload will never become valid; retrying it five times is five wasted minutes and five identical alerts.
  6. Replay republishes with the same job_id, so processed_jobs guarantees no double effect. That property is what makes replay safe enough to use in bulk.
  7. processed_jobs rows expire on TTL; the TTL must exceed the maximum retry window with margin, or a late retry after expiry would re-apply the work.
  8. A dead letter moves pending_review → replayed | discarded, and discarding requires a reason.
  9. High-volume successes are metrics, not rows. Writing a row per successful job would make this table the largest in the platform for no benefit.

​
Data (normative)

​
API (normative)

Ops-only in practice: a tenant does not manage our queues.

​
Events (normative)

None on the outbox. Dead letters alert through observability, not through domain events — a consumer of domain events reacting to our queue failures would be a loop waiting to happen.

​
Acceptance criteria (normative)

  1. The same job_id processed twice concurrently produces exactly one effect.
  2. A crash between the work and the fence leaves neither — proven by an induced failure test.
  3. A transient failure retries 5 times with growing intervals, then writes a dead letter.
  4. A schema-invalid payload goes straight to dead letters with zero retries.
  5. Replay of a dead letter with the same job_id produces no second effect.
  6. TTL cleanup removes expired rows and never removes one inside its retry window.
  7. Discarding without a reason is rejected.
  8. Negative: no successful job writes a row beyond its processed_jobs fence.

​
Execution

Single slice, synchronous command. Schema in TS-001; the harness in TS-002 alongside QueuePort. Runbook: ../../runbooks/dead-letter-replay.md.

​
Open questions

​
Changelog

​
Delivery record

Not implemented yet.