DEV Community

thomasmoore5082
thomasmoore5082

Posted on

Pipeline Feature Flags: Stale Cache Polling Explained Through Decision Logs

Consider a proposed page for an e-commerce platform: the nightly catalog pipeline has produced conflicting decisions from feature flags, and the on-call sees a run identifier beside structured logs showing a stale cache and different polling times. A browser preview says the new processing path is enabled; the server-side job used the old path. Searching for errors alone cannot explain that disagreement. Both decisions may have completed successfully against different cached snapshots.

Short answer: Feature flags suit simple gradual rollouts, but polling permits stale reads and temporary client/server mismatches. Search application-side decision logs by pipeline run, flag key, evaluation surface, and refresh time; shorten polling for critical flags, keep longer intervals for low-risk UX flags, and page on actionable freshness violations rather than every disagreement.

Treat signal quality as the design constraint. This note uses a hypothetical catalog run and explicit example thresholds, not incident measurements or vendor guarantees.

How should stale feature flags and polling mismatches reach the on-call?

The earlier signal is an aging flag snapshot at the point where the pipeline commits to a processing path. A mismatch discovered during a browser preview is downstream evidence: it tells the operator that two decisions differ, but it doesn't establish which snapshot each evaluator held, whether either refresh was overdue, or whether the decisions affected the same run. Those are different investigations, and combining them into one alert creates noise before it creates understanding.

Start with the run.

For this example, define a candidate freshness SLO: a critical flag decision at the pipeline boundary should use a snapshot fetched within 60 seconds. That number needs approval from the application owner, because a tolerable delay for a catalog preview may be unacceptable for the processing path that determines the nightly output. Record the evaluation time as well as the last successful refresh time. A recent log timestamp proves that a log was emitted recently; it does not prove the flag value was fresh.

Now work backward from the page. Search the run's decision records, compare the server and browser surfaces, and inspect their refresh ages before inspecting the returned Boolean. If both snapshots satisfy their freshness target yet disagree, polling-based eventual consistency remains a possible explanation: the two refreshes can straddle an update. If the server snapshot exceeds the target, investigate its refresh behavior first. Neither finding establishes who changed the flag. Preserve change evidence separately when that question matters.

A missing pipeline run is a separate signal. Searching decision records cannot distinguish a job that never started from a job whose logging failed, so use an expected-run monitor such as Healthchecks for that check. Keep its notification separate from a flag mismatch.

Change the evidence before changing the polling interval

Infrai exposes a plain REST API with no SDK to install or client library version to maintain, while one key and one bill cover flags and logs so the pipeline owner can avoid separate provider credentials and invoice reconciliation for those integrations.

The following Go program fetches a flag snapshot and emits an application-owned JSON record to standard output. It is a minimal diagnostic collector, not a replacement for recording the actual application decision. Set INFRAI_BASE_URL to the deployment's API origin without a trailing slash, INFRAI_API_KEY to a credential supplied by the environment, and PIPELINE_RUN_ID to the run being investigated. Save the program as main.go and run it with the standard Go toolchain.

The request method and route below are explicit. The response is preserved as JSON without guessing undocumented flag-response fields; the output wrapper belongs to this example, not the provider's API schema.

package main

import (
    "context"
    "encoding/json"
    "fmt"
    "io"
    "log"
    "net/http"
    "os"
    "strconv"
    "strings"
    "time"
)

func fetch(ctx context.Context, base, key string) (json.RawMessage, time.Time, error) {
    client := &http.Client{Timeout: 10 * time.Second}
    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(
            ctx, http.MethodGet, base+"/v1/flags/get_all", nil,
        )
        if err != nil {
            return nil, time.Time{}, err
        }
        req.Header.Set("Authorization", "Bearer "+key)
        resp, err := client.Do(req)
        if err != nil {
            return nil, time.Time{}, err
        }
        data, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        receivedAt := time.Now().UTC()
        if readErr != nil {
            return nil, time.Time{}, readErr
        }
        if resp.StatusCode == http.StatusTooManyRequests && attempt < 3 {
            delay := time.Second * time.Duration(1<<attempt)
            retryAfter := resp.Header.Get("Retry-After")
            if seconds, err := strconv.Atoi(retryAfter); err == nil && seconds >= 0 {
                requested := time.Duration(seconds) * time.Second
                if requested > delay {
                    delay = requested
                }
            } else if deadline, err := http.ParseTime(retryAfter); err == nil {
                if requested := time.Until(deadline); requested > delay {
                    delay = requested
                }
            }
            timer := time.NewTimer(delay)
            select {
            case <-ctx.Done():
                timer.Stop()
                return nil, time.Time{}, ctx.Err()
            case <-timer.C:
                continue
            }
        }
        if resp.StatusCode < 200 || resp.StatusCode >= 300 {
            return nil, time.Time{}, fmt.Errorf("HTTP %d: %s", resp.StatusCode, data)
        }
        if !json.Valid(data) {
            return nil, time.Time{}, fmt.Errorf("response is not valid JSON")
        }
        return json.RawMessage(data), receivedAt, nil
    }
    return nil, time.Time{}, fmt.Errorf("retry budget exhausted")
}

func main() {
    base := strings.TrimRight(os.Getenv("INFRAI_BASE_URL"), "/")
    key := os.Getenv("INFRAI_API_KEY")
    runID := os.Getenv("PIPELINE_RUN_ID")
    if !strings.HasPrefix(base, "https://") || key == "" || runID == "" {
        log.Fatal("set an HTTPS API origin, API key, and pipeline run ID")
    }
    ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
    defer cancel()
    snapshot, receivedAt, err := fetch(ctx, base, key)
    if err != nil {
        log.Fatal(err)
    }
    record := struct {
        Event      string          `json:"event"`
        RunID      string          `json:"run_id"`
        Surface    string          `json:"surface"`
        ReceivedAt time.Time       `json:"received_at"`
        Snapshot   json.RawMessage `json:"snapshot"`
    }{
        Event:      "pipeline_flag_snapshot",
        RunID:      runID,
        Surface:    "server",
        ReceivedAt: receivedAt,
        Snapshot:   snapshot,
    }
    if err := json.NewEncoder(os.Stdout).Encode(record); err != nil {
        log.Fatal(err)
    }
}
Enter fullscreen mode Exit fullscreen mode

That fetch is evidence of what the diagnostic collector received. It is not proof of what a worker used earlier, and it says nothing about a browser's cache. To make the production record useful, capture the selected flag key, returned value, evaluation time, last successful refresh time, and configured polling interval at the application boundary where the processing path is chosen. Capture value and refresh time from the same cache snapshot. Reading the value first and the timestamp after a concurrent refresh can produce a convincing but false account of freshness. Emit the decision record at path selection rather than on every later use of the cached Boolean; that placement keeps the evidence tied to the decision that can change the nightly output.

For example, a 75-second refresh age breaches the proposed 60-second budget even if the configured polling interval is 30 seconds.

A configuration value isn't an observation.

The full snapshot output is appropriate only for a controlled diagnostic sink: flag values may contain information that should not enter a broadly accessible log stream. For routine collection, retain the relevant decision fields under the team's access and retention policy. For high-risk releases, keep exposure events in the application's analytics or logs layer so the investigation can reconstruct which variant was actually used.

Buy the storage, own the question

Do not confuse the flag interface with the entire operational workflow. The comparison below separates real candidate products from the obligations the team must verify; it makes no feature-parity, latency, or benchmark claim. The axis is useful signal per operational obligation, not a price ranking.

Candidate Buy-versus-build decision for this pipeline Boundary to verify before selecting it
Amazon CloudWatch Logs Evaluate buying the log-storage and search layer while building the decision record and freshness rule in the application. Confirm that the deployed collection and search setup preserves the run identifier and both timestamps; AWS documents log-ingestion billing separately.
Grafana Loki Evaluate it as the log-search candidate where the platform team wants to control how pipeline records enter the logging workflow. Decide who owns deployment, capacity, retention, and alert delivery for the chosen setup; a product name does not assign those duties.
Datadog Evaluate buying the operational workflow when the team wants to assess log investigation alongside its existing monitoring responsibilities. Replay the same run-level investigation and verify collection, access, and notification behavior in the intended deployment.

These are conditional selection criteria, not claims that one backend automatically yields cleaner signals. CloudWatch deserves consideration when an AWS logging workflow fits the platform roadmap; Loki deserves consideration when its logging workflow fits the team's operating model; Datadog deserves consideration when consolidating the operational workflow fits the team's requirements. Reject any candidate if a replay cannot connect a freshness violation to the processing decision that needs intervention.

The application work remains substantial with every option. Someone must define exposure, distinguish server and browser evidence, and decide what action a freshness breach authorizes. Buying a backend does not settle those semantics.

Self-hosting transfers additional capacity and on-call responsibilities to the platform team, so evaluate that choice against the roadmap rather than treating infrastructure ownership as free.

A tighter interval can make the wrong page louder

Capacity planning starts with evaluator count, not the number of flags. Under an illustrative policy with 120 independent evaluators making one refresh request each every 30 seconds, the average refresh load is 4 requests per second; at 120 seconds it is 1 request per second. Those are arithmetic estimates under the stated assumptions. They exclude retries, startup bursts, and request batching, and they say nothing about a provider's measured capacity.

Use short polling selectively for critical flags and longer polling for low-risk UX flags. Lowering every interval can increase request volume while leaving the evidence gap intact, especially when the on-call still cannot tell whether the browser preview and the nightly worker evaluated the same logical change. Spread refresh starts where practical and reserve capacity for the case where many evaluators refresh together.

Then replay the proposed rule against the decision records. A disagreement alert pages for every differing Boolean even when both snapshots are inside the accepted freshness window; a freshness alert pages when a relevant decision exceeds its agreed budget, with the run and evaluation surface attached. Neither rule should invent urgency. If the nightly output can safely wait for verification, route the finding for investigation instead of waking someone immediately.

False positives consume the same on-call attention needed for real pipeline failures. Set the threshold too tightly and normal polling offsets become pages; set it too loosely and the team accepts a longer stale-decision window. Choose that trade-off explicitly with the output owner, then measure how many notifications lead to a concrete intervention. A shorter polling interval earns its place only when it improves that decision.

References

The references below identify the logging alternatives and the complementary expected-run check. The example's timestamps, thresholds, field names, and capacity assumptions are local design choices.

Further reading

Top comments (0)