> ## Documentation Index
> Fetch the complete documentation index at: https://internal.softcrum.com/llms.txt
> Use this file to discover all available pages before exploring further.

# Dual-Rail Dispatch and Workers

> After this ships, a tenant can blast a million emails and another tenant's login codes still arrive in seconds.

## Context

This is the feature that exists because of a specific research finding: **Resend's rate limit is per
team, applied across every API key** — so every tenant shares one budget. Without isolation, one
marketing blast starves every other tenant's OTPs, invitations and receipts. A platform where one
customer's campaign breaks another customer's login is not multi-tenant in any sense that matters.

ADR-015's answer is two rails per channel with strict priority, plus token buckets at two levels:
a **global** bucket per provider matching the provider's real limit, and a **per-tenant** bucket for
fairness among tenants. The global bucket is the one that reflects physics; the per-tenant bucket is
the one that reflects fairness, and both are needed.

## Scope *(normative)*

* Two queues per channel: `notif.{channel}.transactional` and `notif.{channel}.marketing`.
* Worker harness on the shared idempotency and retry infrastructure (FS-CORE-0007).
* Token buckets in Upstash: global per provider, and per tenant.
* Marketing workers yielding when transactional depth crosses a threshold.
* Batch dispatch for email, using the provider's batch endpoint.
* Backpressure and observable queue depth per rail.

## Non-scope *(normative)*

* The adapters themselves — FS-MSG-0006. Dispatch decides when and how fast; adapters deliver.
* Deciding whether to send — FS-MSG-0001.
* `QueuePort` and the generic consumer harness, which are `core` and TS-002.

## Behaviour *(normative)*

1. Every channel has **exactly two rails**. Category maps to rail: `transactional` and `product` to
   the transactional rail, `marketing` to the marketing rail.
2. **Transactional always outranks marketing.** When transactional depth crosses the threshold,
   marketing workers **yield** — they finish the job in hand and stop pulling until depth recovers.
   Yielding, not draining: a marketing worker that dies loses its progress.
3. Two token buckets, both consulted: a **global** bucket per provider matching the provider's real
   limit, and a **per-tenant** bucket for fair use. The global one is physics; the per-tenant one is
   fairness.
4. On a provider `429`, **back off, never drop**. A dropped marketing message is a lost campaign; a
   dropped transactional message is a customer who cannot log in.
5. Email blasts use the **provider's batch endpoint exclusively** (100 per request for Resend).
   Sending a blast one request at a time exhausts the global bucket for everyone.
6. **Contacts are never synced to the provider's audience feature.** Our store is the source of
   truth; a copy there is a second consent record we cannot govern.
7. Every worker is idempotent through `core.processed_jobs`, and exhausted retries land in
   `core.dead_letters` with the send id attached.
8. Queue depth per rail is a published metric with an alert. A transactional rail growing is an
   incident, not a statistic.
9. FORBIDDEN: a marketing job on the transactional rail · bypassing the buckets for an "urgent" send
   · dropping a message on 429.

## Data *(normative)*

| Table                          | Key invariants                                                                                                                                              |
| ------------------------------ | ----------------------------------------------------------------------------------------------------------------------------------------------------------- |
| `messaging.rate_limit_buckets` | configuration only: (`scope` global\|tenant, `provider`, `tenant_id` nullable); `capacity`, `refill_per_second`; the live counters are in Upstash, not here |

## API *(normative)*

| Endpoint                          | Class      | Permission              | Budget    |
| --------------------------------- | ---------- | ----------------------- | --------- |
| `GET /v1/messaging/queues/health` | Management | `messaging.queues.read` | p95 \<1 s |

Ops-facing. A tenant does not tune our rate limits; they see their own throughput in the console.

## Events *(normative)*

None. Dispatch is mechanism; the send events (FS-MSG-0003) are the record.

## Acceptance criteria *(normative)*

1. **The `blast-vs-otp` k6 profile**: a marketing blast at full throughput while a second tenant's
   transactional sends keep their latency budget. This is the profile that proves ADR-015 holds, and
   it is the acceptance criterion the whole feature exists for.
2. Marketing workers yield when transactional depth crosses the threshold, and resume after.
3. A provider 429 causes backoff and zero dropped messages.
4. Email blasts use the batch endpoint; a single-message-per-request path does not exist.
5. One tenant exhausting its per-tenant bucket does not consume another tenant's share.
6. A duplicate job produces one send.
7. Exhausted retries land in dead letters with the send id attached.
8. **Negative:** no code path writes contacts to the provider's audience feature.

## Execution

Asynchronous pipeline. Workers in `backend/workers`, one deployment per ADR-015's isolation
argument. Buckets in Upstash, sharing the primitives used for API rate limiting.

## Open questions

| # | Question                                                                                               | Decides | By              |
| - | ------------------------------------------------------------------------------------------------------ | ------- | --------------- |
| 1 | Transactional depth threshold at which marketing yields — a fixed number, or a function of throughput? | Daniel  | before approval |
| 2 | Do we request a Resend limit raise before G1, and to what number? (deliverability runbook)             | Daniel  | before G1       |

## Changelog

| Version | Date       | Change        | Why | Author                 |
| ------- | ---------- | ------------- | --- | ---------------------- |
| 0.1.0   | 2026-08-17 | Initial draft | —   | daniel + claude-opus-5 |

## Delivery record

*Not implemented yet.*