> ## Documentation Index
> Fetch the complete documentation index at: https://internal.softcrum.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Job Infrastructure — Processed Jobs and Dead Letters

> After this ships, no worker in the platform can double-apply an effect, and no failure can vanish without a record.

## Context

Every queue is at-least-once. A worker will process the same job twice — after a timeout, a deploy,
a provider retry — and in a system that awards points, that is money. Idempotency is not a quality
attribute here; it is the correctness condition, which is why DEC-C6 made `processed_jobs`
mandatory for every consumer rather than a per-worker choice.

The second half is DEC-C5's evidence guarantee: every failure and every exhausted retry is
**persisted**, not merely logged. A log line rotates away; a `dead_letters` row is something an
operator can find, understand and replay. Successes at volume are metrics, not rows — the asymmetry
is deliberate.

## Scope *(normative)*

* `core.processed_jobs`: the dedupe fence, written inside the work transaction.
* `core.dead_letters`: failures with payload, attempts, last error and review state.
* The consumer harness: idempotency gate, retry with backoff and jitter, dead-letter writer.
* TTL cleanup for `processed_jobs`.
* BetterStack alerting on new dead letters.
* Replay from Softcrum Ops, republishing with the same `job_id`.

## Non-scope *(normative)*

* `QueuePort` itself and its adapters — TS-002.
* Notification rails — those are `messaging`, though they use this harness.
* A UI for replay — `frontend/ops`.

## Behaviour *(normative)*

1. A consumer checks and inserts `processed_jobs` **inside the same transaction as its work**.
   Outside it, a crash between the two leaves the job marked done with nothing done.
2. A duplicate `job_id` acks and skips. Silently, with a metric — a duplicate is normal operation,
   not an error.
3. Retry policy: **5 attempts, exponential backoff with jitter**, base 30 s, cap 1 h. Overriding it
   per queue requires a rationale in that module's spec.
4. Exhausted retries and hard failures write a `dead_letters` row with the full payload, attempt
   count, last error and `correlation_id`, and alert BetterStack.
5. **Poison messages skip retries entirely.** A schema-invalid payload will never become valid;
   retrying it five times is five wasted minutes and five identical alerts.
6. Replay republishes with the **same `job_id`**, so `processed_jobs` guarantees no double effect.
   That property is what makes replay safe enough to use in bulk.
7. `processed_jobs` rows expire on TTL; the TTL must exceed the maximum retry window with margin,
   or a late retry after expiry would re-apply the work.
8. A dead letter moves `pending_review → replayed | discarded`, and discarding requires a reason.
9. High-volume successes are **metrics, not rows**. Writing a row per successful job would make
   this table the largest in the platform for no benefit.

## Data *(normative)*

| Table                 | Key invariants                                                                                                                                                                                                   |
| --------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `core.processed_jobs` | PK (`job_id`, `consumer`); `tenant_id`; `processed_at`; `expires_at` for TTL cleanup                                                                                                                             |
| `core.dead_letters`   | `job_id`, `queue`, `payload` JSONB, `attempts`, `last_error`, `status` pending\_review\|replayed\|discarded, `discard_reason` nullable, `correlation_id`, `created_at`; append-only except the status transition |

## API *(normative)*

| Endpoint                                  | Class      | Permission                  | Budget    |
| ----------------------------------------- | ---------- | --------------------------- | --------- |
| `GET /v1/core/dead-letters`               | Management | `core.dead_letters.read`    | p95 \<1 s |
| `POST /v1/core/dead-letters/{id}/replay`  | Management | `core.dead_letters.replay`  | p95 \<1 s |
| `POST /v1/core/dead-letters/{id}/discard` | Management | `core.dead_letters.discard` | p95 \<1 s |

Ops-only in practice: a tenant does not manage our queues.

## Events *(normative)*

None on the outbox. Dead letters alert through observability, not through domain events — a
consumer of domain events reacting to our queue failures would be a loop waiting to happen.

## Acceptance criteria *(normative)*

1. The same `job_id` processed twice concurrently produces exactly one effect.
2. A crash between the work and the fence leaves neither — proven by an induced failure test.
3. A transient failure retries 5 times with growing intervals, then writes a dead letter.
4. A schema-invalid payload goes straight to dead letters with zero retries.
5. Replay of a dead letter with the same `job_id` produces no second effect.
6. TTL cleanup removes expired rows and never removes one inside its retry window.
7. Discarding without a reason is rejected.
8. **Negative:** no successful job writes a row beyond its `processed_jobs` fence.

## Execution

Single slice, synchronous command. Schema in TS-001; the harness in TS-002 alongside `QueuePort`.
Runbook: [`../../runbooks/dead-letter-replay.md`](/runbooks/dead-letter-replay).

## Open questions

| # | Question                                                                                 | Decides | By              |
| - | ---------------------------------------------------------------------------------------- | ------- | --------------- |
| 1 | `processed_jobs` TTL — 7 days, or longer to cover a long provider outage?                | Daniel  | before approval |
| 2 | Should a dead letter older than N days without review escalate, as flagged referrals do? | Daniel  | before approval |

## Changelog

| Version | Date       | Change        | Why | Author                 |
| ------- | ---------- | ------------- | --- | ---------------------- |
| 0.1.0   | 2026-08-17 | Initial draft | —   | daniel + claude-opus-5 |

## Delivery record

*Not implemented yet.*