DEV Community

WilhelmKnight8435
WilhelmKnight8435

Posted on

Per-User Notification Scheduling: Queue Delay, Cron, and HTTPS Webhook Design

Short answer: for per-user scheduled notifications that may outlive a queue delay limit, keep the reminder as durable application state, enqueue only a short rolling horizon, and use cron to reconcile missed promotions rather than to perform delivery inline.

This is the least complicated design that preserves a user-visible promise after a process restart, a reschedule, or a duplicate worker attempt. A delayed message is useful transport; it is a poor system of record. The distinction becomes material as soon as a user can cancel a reminder, change its local time, or ask why it was delivered.

The delivery target should be an at-least-once attempt with an exactly-once business decision. Those are different claims. A distributed queue, an HTTPS endpoint, and an external notification service do not form one transaction, so the useful engineering question is where the durable evidence lives when any boundary is ambiguous.

Transport is not authority.

How should a reminder backend schedule per-user notifications beyond a seven-day queue delay limit?

Start with the constraint, not the scheduling mechanism. A reminder record needs a stable identifier, user identifier, requested local time and IANA time zone, resolved UTC instant, revision, state, and a history of state transitions. The database row represents intent. A queued message represents permission to attempt the current revision of that intent.

That separation makes a long wait ordinary. A promoter periodically claims due records whose fire time has entered the queue's supported delay window, then enqueues an attempt containing reminder_id and revision. For a seven-day queue delay limit, choose a horizon comfortably inside the documented limit and keep that value as one operational setting. The same promoter can run every few minutes; it should be idempotent, batched, and backed by an index over actionable state and due time.

Cron is useful here, but its job is narrow. It invokes promotion and reconciliation. It should not scan for due rows and synchronously send every notification, because one slow downstream call then stretches the lease for the entire run and turns delivery backpressure into database pressure.

Rescheduling is the point that exposes weak designs. Increment the reminder revision in the same transaction that records the new instant. A message for an older revision may still arrive, but the receiver compares its revision with the current row and accepts it as a successful no-op. Cancellation follows the same rule. Nothing needs to be physically removed from a remote queue before the database can make the old work harmless.

Time zones deserve a written product policy. Store the zone supplied by the user and the UTC instant selected under that policy, then define behavior for a repeated local time and for a local time skipped by a clock change. The policy matters for a routine reminder; it matters more for regulated notices, payment prompts, and messages that can expose personal information. Retention, access controls, and audit-log content also need review against the privacy and communications requirements that apply to the service's jurisdictions.

What belongs in the queue, cron fallback, and durable state?

The following split keeps ownership legible during recovery.

Component Owns Must not become
Reminder table User intent, revision, cancellation, and due time A transient worker cache
Queue A bounded, retryable delivery attempt The only record of a future promise
Cron-driven promoter Admission into the queue horizon and repair work The synchronous notification sender
Delivery outbox A committed request to perform an external effect An unaudited best-effort callback

Amazon SQS documents visibility timeout as a processing control: after a consumer receives a message, other consumers do not see it for the timeout period, and the message can become visible again if it is not deleted in time. This is why a receiver must tolerate redelivery; invisibility is not proof that a notification effect happened exactly once. A queue's delay facility has the same character: it is a delivery primitive with a documented operating contract, not a substitute for domain history.

Commit intent before trying to publish work. If publication is interrupted after the database commit, reconciliation can find the visible reminder and retry promotion. If work is published before intent commits, a receiver can observe a message with no durable explanation. The common repair is an outbox row written with the reminder change, followed by a separate publisher that marks the row after successful handoff. The queue handoff can still be repeated, which is expected.

For the recipient, authenticate the public HTTPS webhook before mutating state. TLS protects the connection, but a URL alone is not identity. Verify a signature or token under a documented rotation policy, associate the incoming request with its reminder and revision, and reject credentials that do not satisfy that policy. Keep the idempotency identity independent of the authentication credential so key rotation cannot create a second business effect.

Here is the critical transaction in Go. It records the decision to deliver before any external sender is asked to do work.

package reminders

import (
    "context"
    "database/sql"
    "fmt"
)

type Attempt struct {
    ReminderID string
    Revision   int64
}

func AcceptAttempt(ctx context.Context, db *sql.DB, attempt Attempt) error {
    tx, err := db.BeginTx(ctx, &sql.TxOptions{Isolation: sql.LevelSerializable})
    if err != nil {
        return fmt.Errorf("begin delivery transaction: %w", err)
    }
    defer tx.Rollback()

    var currentRevision int64
    var state string
    err = tx.QueryRowContext(ctx,
        `SELECT revision, state FROM reminders WHERE id = $1 FOR UPDATE`,
        attempt.ReminderID,
    ).Scan(&currentRevision, &state)
    if err != nil {
        return fmt.Errorf("load reminder: %w", err)
    }
    if currentRevision != attempt.Revision || state == "cancelled" {
        return tx.Commit()
    }

    key := fmt.Sprintf("%s:%d", attempt.ReminderID, attempt.Revision)
    result, err := tx.ExecContext(ctx,
        `INSERT INTO delivery_intents (idempotency_key, reminder_id, revision)
         VALUES ($1, $2, $3) ON CONFLICT (idempotency_key) DO NOTHING`,
        key, attempt.ReminderID, attempt.Revision,
    )
    if err != nil {
        return fmt.Errorf("record delivery intent: %w", err)
    }
    inserted, err := result.RowsAffected()
    if err != nil {
        return fmt.Errorf("inspect delivery intent: %w", err)
    }
    if inserted == 0 {
        return tx.Commit()
    }

    _, err = tx.ExecContext(ctx,
        `INSERT INTO notification_outbox (idempotency_key, payload_ref)
         VALUES ($1, $2)`, key, attempt.ReminderID)
    if err != nil {
        return fmt.Errorf("write notification outbox: %w", err)
    }
    return tx.Commit()
}
Enter fullscreen mode Exit fullscreen mode

The outbox worker performs the external send later and stores the downstream reference. Where the downstream interface accepts an idempotency key, reuse this identity. Where it does not, the delivery contract must admit that an ambiguous response can lead to a duplicate attempt; the application can still keep one auditable decision to deliver.

Which scheduling model fits the actual reminder contract?

There is no universal best reminder backend. The right model follows the ownership and mutation requirements, not a feature checklist.

Model Good fit Cost of the choice
Durable table plus rolling queue horizon Reminders are editable, cancellable, and need audit history Requires promotion, an outbox, and reconciliation
Leased database cron worker Low volume and minute-level timing tolerance Polling, leases, and backpressure remain application responsibilities
Durable workflow system A reminder is one wait in branching, long-running work Introduces a workflow programming and operations model
Queue-only delayed work Tasks fit inside the provider's documented horizon and history needs are modest Future intent becomes coupled to transport state

The catch is operational scope. A durable table plus queue is not suitable for disposable, low-volume internal reminders where a few minutes of timing variation is acceptable and nobody needs a delivery audit. A leased cron worker is often the more proportionate choice there. Conversely, a reminder that is really part of a long-lived approval or compensation process can justify a workflow model, because its history and signals are part of the domain rather than incidental machinery.

The work does not end at enqueueing. Track the age of the oldest overdue reminder, promotion lag, outbox age, attempts by outcome, duplicate suppression, and reconciliation repairs. Queue depth alone cannot establish whether the oldest user promise is late. Store an audit event with actor, transition, revision, attempt, timestamp, and trace reference, while keeping message bodies out of routine logs.

This also changes how an operator investigates a complaint. Start with the reminder identifier and reconstruct the transition history: creation or reschedule, the revision selected for promotion, the outbox record, queue publication, receipt, delivery-intent decision, and the downstream reference if a sender was invoked. Each step should have a time and an owner, and the state model should distinguish “not yet eligible,” “eligible but not published,” “published,” and “delivery decision committed.” Collapsing these states into a single boolean makes reconciliation guess at intent. The same history supports a conservative retry policy: a reconciliation job can republish an unfinished outbox record, while the unique delivery-intent key stops a second committed decision for the same reminder revision. Auditability is therefore not a reporting accessory. It is the mechanism that lets an on-call engineer repair a boundary failure without deciding, from incomplete queue observations, whether a user should receive a second message.

Can a public HTTPS webhook remain correct during retries and rollout?

Yes, provided the endpoint is only a bounded step in a recoverable protocol. Classify invalid authentication and malformed payloads as terminal, treat cancellation and stale revisions as successful no-ops, and retry failures whose contract permits a later attempt. A handler should acknowledge only after the delivery-intent transaction commits. Do not let a process-local lock or an in-memory timer carry a correctness guarantee across restarts.

Before rollout, test duplicate receipt, cancellation racing with dispatch, rescheduling after promotion, a process exit after committing an outbox row, repeated and skipped local times, and a callback timeout after acceptance. Use a controllable clock for unit tests and the actual queue contract for integration tests. Then rehearse recovery: pause promotion, let the reconciliation query find the backlog, and verify that current revisions are dispatched without creating a second delivery intent.

Small first. The migration path is to add revision and state history to the existing reminder table, create the actionable-state index, introduce an outbox publisher, and enable the promoter behind metrics. Backfill only active reminders, run reconciliation in observation mode, and make the enqueue horizon explicit in configuration. This order keeps old and new delivery paths explainable during the transition, which is more valuable than a fast cutover when the system carries user commitments.

References

Further reading

Top comments (0)