DEV Community

rasmusberg6592
rasmusberg6592

Posted on

Log-Based Failure Alerts: Poll Error Status and Search Notification API Evidence

A useful page for a healthtech notification service shows the failed delivery stage, a correlation identifier, and the window to investigate. Start there when choosing log-based failure alerts for a Node.js Express API. A page that reports request errors alone can leave the responder inspecting the handler while delivery work has already moved downstream.

Short answer: use structured logs for failure alerts when the service already emits application events, and assign ownership of polling, threshold evaluation, and notification delivery before choosing a backend. For incident reconstruction, the deciding question is whether the responder can connect an accepted request to its eventual delivery outcome. A managed alerting workflow is worth evaluating when the platform team cannot support that checker.

The discussion below uses a proposed notification workflow, not a production incident or a benchmark. Its numbers are planning examples. The objective is an actionable page with enough evidence to reconstruct delivery, rather than a contest over ingestion prices.

How should log-based failure alerts search API error status for delivery evidence?

Consider an illustrative page: delivery failures in the reminder cohort, two failed attempts, one affected notification, and a trace identifier linking the request to its worker events. The responder needs to distinguish a transient attempt failure from a terminal outcome before deciding whether the service is consuming its delivery error budget. Those two counts describe different things.

Follow the handoff.

An accepted API request establishes acceptance at that boundary. It does not establish delivery. Emit structured events with level, route, user, trace_id, and status_code, then carry the correlation identifier through the work that performs delivery. Where span_id is useful, include it as another correlation field. Shared fields let a responder assemble a sequence; they do not, by themselves, provide a distributed tracing UI or a parent-child span tree.

For healthtech, define user as an opaque application identifier and keep patient details out of the example event contract. An opaque identifier still needs a data lifecycle decision. Before admitting production identifiers, establish whether deletion, export, and retention controls satisfy the application's requirements; having a convenient search box does not settle that question.

Define the delivery SLO at the outcome boundary. If a retry produces another failure event for the same notification, the log count increases without necessarily increasing the number of failed notifications. The page should carry both the attempt evidence and the outcome interpretation, with the latter tied to an explicit event contract. Otherwise, a retry policy change can appear to be a reliability regression even when the final delivery result has not changed.

Work backward to the signal that should have fired earlier

The earlier useful signal is a delivery failure associated with its originating request and work item. Searching only request-handler error status misses the distinction between acceptance and completion. Searching only error-level entries also loses the successful events needed to explain a recovery or construct a denominator.

Silence is different.

If expected work never starts, there may be no error event to find. Evaluate a heartbeat tool such as Healthchecks for that missing-execution question, and retain application events for what happened after execution began. Neither a clean search result nor a falling error count proves that notification delivery is healthy.

The checker therefore needs separate outcomes for a successful search with no matching failures, an unsuccessful collection, and evidence that expected work did not run. Treating all three as zero failures is a concrete implementation trap: it lets the detection path disappear behind a reassuring chart. Record checker health and route its own operational failures through an owned response process.

Capacity planning belongs here, before shortening the polling interval. At one check every 60 seconds, a continuously running checker performs 1,440 checks per day before retries. Ten independently polled cohorts would make that 14,400 checks. These are arithmetic examples, not service quotas. Include response volume, delayed-event overlap, retry traffic, and investigation queries in the capacity estimate, because the responder will need search capacity precisely when the checker is busiest. A checker per user is a particularly expensive ownership decision to make accidentally.

Poll frequency also bounds the opportunity to detect an event; it is not an end-to-end detection guarantee. Event arrival, checker execution, evaluation, and notification delivery all consume time. Allocate those stages against the response objective instead of presenting the polling interval as the alert latency.

Change the event contract before automating the verdict

The instrumentation change is small but consequential: record the work item and attempt alongside the shared trace identifier, and give terminal outcomes an explicit meaning. In an illustrative event sequence, one accepted request leads to two failed delivery attempts and one terminally failed notification. The worker must distinguish an attempt failure from the terminal outcome using its actual retry policy. Counting the two error entries as two failed notifications would be wrong.

The following standalone Go collector retrieves log-search evidence without inventing filter parameters or a response schema. Save it as a Go source file and run it with the Go toolchain. Set INFRAI_BASE_URL to the documented HTTPS API base including its version prefix, and INFRAI_API_KEY to your key. The program prints the returned JSON unchanged; interpretation and paging remain separate responsibilities.

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

func run() error {
    base := strings.TrimRight(os.Getenv("INFRAI_BASE_URL"), "/")
    key := os.Getenv("INFRAI_API_KEY")
    parsed, err := url.Parse(base)
    if err != nil || parsed.Scheme != "https" || parsed.Host == "" ||
        parsed.User != nil || parsed.Path != "/v1" ||
        parsed.RawQuery != "" || parsed.Fragment != "" || key == "" {
        return fmt.Errorf("set the HTTPS versioned API base and API key")
    }
    client := &http.Client{
        Timeout: 15 * time.Second,
        CheckRedirect: func(req *http.Request, via []*http.Request) error {
            return http.ErrUseLastResponse
        },
    }
    ctx, cancel := context.WithTimeout(context.Background(), 90*time.Second)
    defer cancel()
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet,
            base+"/logs/search", nil)
        if err != nil { return err }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := client.Do(req)
        if err != nil { return err }
        body, readErr := io.ReadAll(io.LimitReader(resp.Body, (8<<20)+1))
        resp.Body.Close()
        if readErr != nil { return readErr }
        if len(body) > 8<<20 { return fmt.Errorf("response exceeds local cap") }
        if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
            delay := time.Second * time.Duration(1<<attempt)
            retry := resp.Header.Get("Retry-After")
            if seconds, e := strconv.ParseInt(retry, 10, 64); e == nil && seconds >= 0 {
                if seconds > 90 { return fmt.Errorf("retry exceeds collection budget") }
                delay = time.Duration(seconds) * time.Second
            } else if when, e := http.ParseTime(retry); e == nil {
                delay = time.Until(when)
                if delay < 0 { delay = 0 }
            }
            timer := time.NewTimer(delay)
            select {
            case <-ctx.Done():
                timer.Stop()
                return ctx.Err()
            case <-timer.C:
                continue
            }
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return fmt.Errorf("HTTP %d: %s", resp.StatusCode, body)
        }
        if !json.Valid(body) { return fmt.Errorf("invalid JSON response") }
        _, err = os.Stdout.Write(body)
        return err
    }
    return fmt.Errorf("retry budget exhausted")
}

func main() {
    if err := run(); err != nil {
        fmt.Fprintln(os.Stderr, err)
        os.Exit(1)
    }
}
Enter fullscreen mode Exit fullscreen mode

The 15-second request timeout, 90-second collection deadline, four attempts, and 8 MiB response cap are local safeguards, not backend guarantees. Rate-limited requests back off, honoring Retry-After within the collection budget. This read-only operation does not create a duplicate write when retried. Don't interpret an unfiltered response as a complete delivery failure rate: verify selection, completeness, and event interpretation before using results to evaluate an SLO.

Before deploying a checker, settle how it selects a complete window, recognizes repeated records, and distinguishes new terminal outcomes from failures it has already paged on. Keep a stable incident identity across overlapping searches. Confirm the backend's documented request and response schemas before adapting this local example to ingestion or search; do not assume that application field names are accepted query parameters.

The evidence has to survive the handoff from detection to investigation. Include the evaluation window and relevant identifiers in the page so the responder can repeat the search, inspect surrounding successful events, and explain why the threshold crossed. A count without that context is a weak incident artifact.

Buy versus build: who owns the evidence-to-page workflow?

For a platform roadmap, the purchase decision is about operational responsibility and reconstruction depth. Compare the same failed-delivery exercise across products, including a request that was accepted, repeated delivery attempts, and an execution that never began. An impressive error search does not answer all three.

Option Buy versus build choice Fit and boundary to evaluate
Datadog Evaluate its log-monitor workflow as the managed baseline. Favor this evaluation when managed thresholding and the page workflow matter; require a demonstration that the alert leads back to the notification's delivery evidence.
Grafana Loki with Grafana Alerting Evaluate log storage and alert evaluation together, with explicit deployment ownership. A candidate when control over the logging stack justifies operating the components; budget for correlation conventions and the on-call work of the chosen deployment.
Sentry Crons Evaluate scheduled-execution monitoring against the service's job boundary. Relevant to whether expected work ran; separately demonstrate how worker evidence establishes the final notification outcome.
Healthchecks Evaluate heartbeat monitoring for missing execution. Relevant to silent jobs; a heartbeat alone does not reconstruct the failed delivery attempt.
REST log backend with an owned checker Build scheduled searches, evaluation, deduplication, and notification delivery around structured events. Appropriate when the team wants searchable application evidence and accepts responsibility for the complete detection path.

Infrai fits the final row with one REST API, no SDK required, and a single key with unified billing for logs and metrics, reducing separate credential configuration if the platform team uses both during delivery investigations. Any language that can send HTTP requests can use the API. Its public, self-describing discovery surface requires no key, which helps the team review schemas before wiring production credentials into the checker.

That fit has an operational boundary. Choose this polling design only if the platform team can own threshold evaluation and notification delivery, accept reconstruction through log identifiers, and establish that user deletion and bulk export requirements are met by the selected system. If the requirement is an interactive span tree, evaluate a tracing workflow explicitly. If it is source-map resolution, crash symbolication, or session replay, include those as separate acceptance criteria rather than treating log search as evidence that they are covered.

The table is an evaluation plan, not a performance ranking. Require each candidate to preserve enough event context for the same reconstruction exercise, then compare managed operational responsibility against the burden of operating the selected components. Include migration in that review: application-owned identifiers help keep the evidence contract understandable, but extraction and downstream access still need verification before committing a log archive to a platform.

Close the loop without turning every retry into a page

A threshold that counts attempts can page twice for the one synthetic notification above. A threshold that waits only for terminal failures may suit the delivery SLO, but it can hide deteriorating retry behavior until the delivery outcome is decided. Keep attempt pressure available for investigation and capacity planning while giving the paging rule a clearly stated relationship to final outcomes.

Use the error budget.

Choose thresholds and evaluation windows from the service's delivery objective and actual event volume, then review their behavior against known outcomes before enabling paging. A low-volume reminder cohort and a busy cohort need separate consideration; neither a universal count nor an unsupported percentage makes a defensible default. Deduplication, delayed events, and recovery semantics belong in that review because they change how often a responder is interrupted.

The false-positive cost is the interruption plus the reconstruction work required to dismiss it. Set the threshold too low and repeated attempts repeatedly consume on-call attention; set it too high and terminal delivery failures accumulate before anyone investigates. The purchase decision should leave a named owner for that trade-off. The useful final artifact is a page that can explain itself from application evidence.

Further reading

References for evaluating the workflows and terminology discussed above:

Top comments (0)