Skip to content
Webitops

Notes

Reserve the quota inside the transaction

Sending a metered notification from a queue worker has three distinct failure modes, and retry logic fixes none of them. Modelling delivery as a row — claimed and budgeted in one transaction — fixes all three.

5 min read Laravel, Queues, Idempotency

Here is a piece of code that appears in a great many applications, and is wrong in a way that takes months to notice.

// Don't do this.
public function updated(Ticket $ticket): void
{
    if ($ticket->wasChanged('status_id')) {
        $ticket->business->decrement('email_credits');
        SendStatusEmail::dispatch($ticket);
    }
}

It sends an email when a ticket changes status and charges the tenant for it. It has three separate bugs, and only one of them is the one people usually look for.

Three failure modes, not one

Double-send. The job runs, times out after handing off to the mail provider, and is retried. The customer gets two “your repair is ready” messages. The usual fix — a sent_at column checked at the top of the job — narrows the window but does not close it: two workers can both read null before either writes.

Phantom quota burn. The credit is decremented, then the send fails terminally on an invalid address. The tenant paid for nothing. Or worse: the decrement and the send are in different transactions, one commits and the other does not, and the ledger and reality part company permanently.

Lost delivery on rollback. dispatch() inside a transaction hands the job to the queue immediately. If the surrounding transaction then rolls back, a worker picks up a job pointing at a status change that no longer exists. On a fast queue this races even without a rollback: the worker can start before the transaction commits and find nothing there.

Retry configuration addresses none of these. They are not transport problems.

Model the delivery, not the send

The fix is to stop thinking of sending as an action and start thinking of it as a claim that gets recorded, budgeted, and only then executed.

DB::transaction(function () use ($ticket, $status, $channel) {
    // 1. The row IS the claim. The unique index is the concurrency control.
    try {
        $delivery = NotificationDelivery::create([
            'ticket_status_id' => $status->id,
            'channel'          => $channel->name(),
            'state'            => DeliveryState::Pending,
        ]);
    } catch (QueryException $e) {
        if (! $this->isUniqueViolation($e)) {
            throw $e;
        }
        return; // Someone else already claimed this exact send. Nothing to do.
    }

    // 2. Budget it in the same transaction. Out of credit => the claim
    //    rolls back too, and there is no orphan row to retry later.
    $this->quotas->reserve($ticket->business, $channel->quotaKey(), $delivery);

    // 3. Dispatch last, and only if the transaction survives.
    SendNotificationDelivery::dispatch($delivery)->afterCommit();
});

Three properties, each fixing one of the bugs above:

  1. The unique index on (ticket_status_id, channel) is the idempotency key. Not a flag that gets checked, a constraint that cannot be violated. Concurrent attempts race for the same key; the loser catches the violation and returns. This is the only version that is actually safe under concurrency, because the database is doing the mutual exclusion rather than your application logic.
  2. The reservation shares the transaction with the claim. Both happen or neither does. There is no window in which one exists without the other.
  3. afterCommit() moves dispatch after the commit. No job can reference a row that was rolled back, and no worker can start before the data it needs is visible.

Terminal and transient are different failures

Once delivery is a row with a state, the worker needs to say why it failed, and there are exactly two answers that matter:

try {
    $channel->send($delivery);
    $delivery->markSent();
} catch (TransientDeliveryException) {
    throw;                       // let the queue retry; reservation stands
} catch (TerminalDeliveryException $e) {
    $delivery->markFailed($e);
    $this->quotas->release($delivery);   // give the credit back
}

A provider timeout or a 5xx is transient: retry, keep the reservation held. An invalid phone number or a rejected template is terminal: stop, mark it failed, and release the budget.

Collapse those into a generic catch-and-retry and you get a queue that spends the night retrying an address that will never be valid while the tenant’s allowance stays locked. This distinction is worth encoding as two exception types rather than a boolean, because the classification lives with the channel adapter that actually understands the provider’s error codes.

You still need a reconciler

The uncomfortable case: a worker dies between reserving and sending. No exception is thrown, so nothing is classified. The row sits Pending forever with budget held against it.

No amount of transactional care removes this — the process can always vanish. So a scheduled command sweeps deliveries left pending beyond a threshold and either re-queues or fails them.

That sweeper is only safe to run because of the idempotency work. Re-queuing a delivery that actually did send is harmless when the send is claimed by a unique row. In a design where sending is an action rather than a record, a reconciler is a double-send generator.

What it costs

An extra table with a unique index on a hot write path, a scheduled command, two exception types, and tests that have to exercise concurrency rather than a happy path.

What you get is that the awkward questions have boring answers. Can a customer get the same message twice? No — the constraint prevents it. Can a tenant be billed for a send that never happened? No — terminal failures release the reservation. What if we deploy mid-send? It is retried, and finds the work already claimed.

Those are good answers to have ready when the customer asking is the one being billed.