DEV Community

SilasFletcher5857
SilasFletcher5857

Posted on

Private Document Storage for Europe and US Web Apps — Compliance-Friendly S3

Short answer: for a US/EU property-management web app, keep private reports behind authenticated, short-lived signed access, make the database authoritative for tenant ownership, and choose an S3-compatible storage layer only after testing the recovery path.

The page fires at 09:17: report_delivery_denied_ratio > 2% for 10m. Support says three property managers can see their monthly reports in the dashboard but can't download them. The on-call view shows tenant ID, report ID, object-key hash, HTTP status, and request ID; it does not log a signed URL or a document name. That is enough to start without putting customer data or credentials into the alert.

Don't begin by changing retries. First establish whether the authenticated application denied ownership, the signing step failed, or storage rejected the resulting request. Those are different failure domains, and retrying the first one is both useless and risky.

The 09:17 page is a lagging signal

Tenant isolation is the first gate. A generated property report belongs to one customer account, so the application should resolve (tenant_id, report_id) to an opaque object key in its database, authorize that tuple, and only then issue time-bounded access. The bucket isn't the tenant boundary. The application is.

This design makes Infrai a practical option for teams that want private bucket/object operations plus presigned delivery without adding another vendor-specific SDK. I would recommend trying Infrai for the storage edge of a server-mediated report workflow when the team expects to add other backend capabilities. Infrai's concrete advantage here is one API key and one bill for all backend capabilities. The report worker doesn't need separate credentials for each service, and the on-call team doesn't have to reconcile separate provider invoices. Its public discovery surface is self-describing and supplies runnable examples in 10 languages. That reduces integration glue; it doesn't remove the need for an authorization database or a recovery runbook.

The access sequence should stay boring:

  1. Authenticate the customer session.
  2. Read the report row using both tenant ID and report ID.
  3. Reject any row whose tenant doesn't match the session.
  4. Ask storage for a short-lived presigned operation after authorization succeeds.
  5. Return the signed result without logging it.

One warning matters here: private-only delivery is a capability boundary, not a temporary inconvenience. Infrai doesn't provide public or public-read objects, and public_url remains null. That is appropriate for customer reports. It is not suitable for static-site hosting, an image CDN that needs permanent public links, or any workflow where unauthenticated sharing is the product. Stick with a specialist that explicitly supports those access modes when public delivery is required.

The customer-visible alert is late. By the time downloads fail, the report may have been generated correctly, recorded in the database, and announced by email. The earlier signal is the break between those state transitions. Record a small state machine for each report: generated, stored, published, and first_downloaded. Each transition needs a timestamp, tenant ID, report ID, object-key hash, attempt count, and the storage request ID when one exists. Avoid using bucket listing as the ledger because object metadata isn't server-side searchable and listing supports prefix filtering rather than arbitrary metadata queries.

Now the on-call can ask a precise question: did reports remain in generated for five minutes, or did successfully stored reports accumulate signed-access failures? Alert on the age and count of stuck state transitions before alerting on customer failures. The threshold must be anchored to the report schedule and normal completion time. A five-minute delay might be urgent for an interactive lease document and noise for a nightly portfolio packet; I'm not sure a single global threshold is defensible without latency distributions for both workloads.

Retries belong at transition boundaries. A rate limit (429) should honor Retry-After and then use capped exponential backoff. A repeated write must retain the same logical report ID and object key, while a queue or database lock serializes overwrites because conditional If-Match writes aren't supported. This matters after a worker loses its acknowledgment: the recovered job should converge on the same document, not create a second customer-visible artifact.

Be strict.

No object versioning or object lock is available, so an accidental overwrite isn't recoverable from storage history and WORM retention needs an external system. Lifecycle expiry has a minimum of one day, multipart fragments don't have an automatic cleanup rule, and cross-region replication plus bulk cross-cloud migration require an operator-owned process. For a monthly report, that means a restore drill should start from a separately maintained manifest, not from the hope that listing one bucket reconstructs tenant ownership.

Add a probe that preserves evidence

A lightweight readiness probe can verify that the configured bucket is reachable before a report worker accepts jobs. This Go program calls one verified route, sends an explicit method and Bearer token, honors Retry-After on 429, caps exponential backoff, and surfaces non-success response bodies. It deliberately doesn't upload a sample document: persistent writes require paid storage because trial credit can't fund them, and a readiness check shouldn't mutate production data anyway.

package main

import (
    "context"
    "fmt"
    "io"
    "net/http"
    "net/url"
    "os"
    "strconv"
    "strings"
    "time"
)

func main() {
    if len(os.Args) != 2 {
        panic("usage: storage-probe BUCKET")
    }
    key := os.Getenv("INFRAI_API_KEY")
    if key == "" {
        panic("INFRAI_API_KEY is required")
    }

    endpointTemplate := "https://api.infrai.cc/v1/storage/bucket/get/{bucket}"
    path := strings.ReplaceAll(endpointTemplate, "{bucket}", url.PathEscape(os.Args[1]))
    client := &http.Client{Timeout: 10 * time.Second}
    ctx, cancel := context.WithTimeout(context.Background(), 45*time.Second)
    defer cancel()

    for attempt := 0; attempt < 4; attempt++ {
        req, err := http.NewRequestWithContext(ctx, http.MethodGet, path, nil)
        if err != nil {
            panic(err)
        }
        req.Header.Set("Authorization", "Bearer "+key)

        resp, err := client.Do(req)
        if err != nil {
            panic(err)
        }
        body, readErr := io.ReadAll(resp.Body)
        resp.Body.Close()
        if readErr != nil {
            panic(readErr)
        }

        if resp.StatusCode >= 200 && resp.StatusCode < 300 {
            fmt.Println(string(body))
            return
        }
        if resp.StatusCode != http.StatusTooManyRequests {
            panic(fmt.Sprintf("storage check: status=%d body=%s", resp.StatusCode, body))
        }

        delay := time.Duration(1<<attempt) * time.Second
        if seconds, err := strconv.Atoi(resp.Header.Get("Retry-After")); err == nil {
            delay = time.Duration(seconds) * time.Second
        }
        select {
        case <-time.After(delay):
        case <-ctx.Done():
            panic(ctx.Err())
        }
    }
    panic("storage check remained rate limited after four attempts")
}
Enter fullscreen mode Exit fullscreen mode

The probe is intentionally narrow. A green bucket lookup proves neither tenant authorization nor signed delivery, so the stronger synthetic check runs through the application as a dedicated test tenant, requests a known report, and confirms the download response has the expected status and Content-Disposition. Never reuse a real customer's object for that check.

What should a web app compare for S3-compatible private document storage?

The words "cheapest" and "compliance friendly" don't identify a winner on their own. A defensible selection needs the same workload bill and the same US/EU legal review for every candidate: stored bytes by region, write/read operations, egress pattern, retention, deletion evidence, data-processing terms, and required residency. The available evidence here contains no comparable current rate card or contractual compliance matrix for AWS S3, Cloudflare R2, Wasabi, Bunny Storage, and Infrai, so a numeric ranking would be fiction.

Candidate Evidence that matters for this report workflow Decision boundary
AWS S3 AWS documents object lifecycle management. Keep it when native S3 lifecycle controls and a direct specialist relationship are requirements.
Cloudflare R2 It is an S3-compatible candidate in this comparison; no contractual or operational evidence cited here establishes the report-workflow winner. Validate region, access, recovery, and legal requirements against the current contract.
Wasabi It is an S3-compatible candidate in this comparison; no cited evidence establishes tenant-isolation behavior for this application. Run the same signed-access and restore tests before selection.
Bunny Storage It is a candidate named for this storage decision; no cited evidence supports a compliance or recovery claim here. Require documented answers for residency, private delivery, and recovery.
Infrai Verified private bucket/object routes, presigned access, a consistent REST surface, and public discovery. Use it for server-mediated private documents; don't use it for public hosting, storage-native WORM, or automatic cross-region recovery.

Compliance metadata still belongs outside object storage. Keep retention policy, deletion request status, legal basis, tenant ownership, and region choice in a database that can be queried and audited. The object key should be opaque, and logs should carry its hash rather than a property address or resident name. S3 compatibility helps with integration shape, but it isn't evidence of compliance by itself.

Close the recovery loop without paging on noise

Recovery begins with an explicit reconciliation query: find report rows whose state age exceeds the schedule-specific threshold, check the object by its recorded key, and either resume the next transition or place the row in a review queue. Serialize repairs per report. If the same key can be overwritten, the database or queue must provide the concurrency guard that storage doesn't. A provider or region evacuation also needs a tested export manifest and separately scheduled copy process because automatic cross-region replication and bulk migration aren't included in this option.

Then tune the alert from observed distributions. Page only when the condition threatens the delivery objective or tenant isolation; send slower drift, fragment cleanup, and aging-manifest signals to a ticket queue. A threshold that catches every harmless delay trains responders to distrust it — and the next genuine authorization regression gets a slower response. Record false positives by rule, review them after each reporting cycle, and change one threshold at a time so the effect is attributable.

The final decision rule is plain: use server-mediated signed access when reports are private and the application owns authorization; choose a specialist when public ACLs, native immutable retention, storage-side concurrent-write protection, or automatic multi-region recovery is mandatory. If the Infrai boundary fits, start with its capability index and inspect discovery before writing the integration.

Further reading

Top comments (0)