DEV Community

Cover image for Tail sampling rules in the OpenTelemetry Collector: cost and pitfalls
Span Observer
Span Observer

Posted on Originally published at span.observer

Tail sampling rules in the OpenTelemetry Collector: cost and pitfalls

Your second-tier Collector keeps restarting, or your trace bill refuses to fall: either way, it is your tail sampling rules that need another look, well before you touch the traffic. Every rule you add changes what the processor holds in memory, what it discards and what reaches the backend. This is not a configuration detail. It is a budget you have to arbitrate.

This article starts from the operating decision rather than the list of policies. How much memory do you grant to traces « in flight »? In what shape do you combine the rules? Which traps wait after go-live? The basics are already covered in the OpenTelemetry tail sampling tutorial, and trace ID routing in the load balancing article. Here we size and tune.

The facts come from the tail_sampling processor README (opentelemetry-collector-contrib repository) and the sampling page of opentelemetry.io, both read on 11 November 2026. Everything happens in the Collector, before the OTLP export, so the reasoning holds for Dynatrace SaaS, for Dynatrace Managed and for any other backend.

What a rule costs: memory, first of all

The README flags the processor with a statefulness warning: it keeps spans in memory until it has decided. The price of a rule set is therefore read in memory long before CPU. Three settings drive it.

  • decision_wait (30 s by default): how long the processor waits before evaluating a trace. At constant throughput, doubling it doubles the number of traces held.
  • num_traces (50,000 by default): the size of the circular buffer. When a new trace arrives and the buffer is full, the oldest one is evicted, even if it has not been evaluated.
  • maximum_trace_size_bytes: a trace larger than this is dropped immediately. It protects you from monster traces.

The sizing rule that follows is plain arithmetic, not an official formula: num_traces must be at least the number of new traces per second, per instance, multiplied by decision_wait. Here is a worked example, using assumptions I picked myself. Replace them with your own measurements.

Terminal: a Python script computes 2000 new traces per second per instance, so 60,000 traces in flight at decision\_wait 30 s, which exceeds the default num\_traces of 50,000

In this example the default is too small: the buffer evicts traces before a decision, and the README names the metric that reveals it, sampling_trace_dropped_too_early. Lowering decision_wait to 10 s brings the need down to 20,000 traces. I have not measured the real in-memory footprint of a span inside the processor. The megabytes shown are raw data under assumption, and real consumption will be higher. Measure it on your own traffic before you set pod limits.

Combining tail sampling rules so they answer each other

The processor evaluates every policy, then settles the outcome with the decision flow in the README. In practice: a drop policy that fires discards the trace, a « sample » policy keeps it, and in every other case the trace is not sampled. What matters is therefore less the order than the nature of each rule.

Three families of rules

  1. Exclusions: drop removes noise, such as health probes. It wins over policies that would otherwise keep the trace.
  2. Safety nets: status_code on ERROR and latency above a threshold keep what the on-call engineer will actually read to cut MTTR.
  3. The cost ceiling: composite shares a maximum number of spans per second between sub-policies. A rate_allocation percentage reserves room for the critical service, and a probabilistic sub-policy takes the rest.

File otelcol-tier2.yaml: tail\_sampling processor with decision\_wait 10s, num\_traces 40000, a decision cache, a drop policy on /health and /ready, a status\_code ERROR rule, a latency rule and a composite policy at 4000 spans per second

This configuration is an example written from the README. span.observer did not run it in a Collector, and the service names, thresholds and paths are made up. It does not replace a test on your own traffic.

Two nuances deserve attention. First, the README recommends adding the probabilistic policy inside the processor when you already use it: the extra cost is negligible, and traces kept by other rules are not thrown away by a second sampler downstream. Second, sample_on_first_match decides as soon as one policy matches. It is faster, but the README advises against combining it with the usetracestate feature gate, because stopping early skews the adjusted counts downstream.

Production pitfalls of tail sampling rules

Spans that arrive too late

A « late » span arrives after the decision. If the decision is still in the buffer, the span inherits it. Otherwise, with no cache, the processor treats it as an unknown trace and opens a new decision_wait window, so one trace can end up half kept. The decision_cache (sampled_cache_size and non_sampled_cache_size, both 0 by default) remembers decided trace IDs. The README advises sizing it well above num_traces.

One Collector, or several

As soon as there are several instances, every span of a trace must reach the same one. The README recommends two layers: the load balancing exporter in front, the processor behind, to isolate failures. Setup details are in the dedicated article. For the processor’s place in a pipeline, see also the Collector contrib components: the README says it must come after processors that depend on context, such as k8sattributes, because it reassembles spans into new batches.

Rate limits under several shards

With num_shards above 1, num_traces and the per-second limits are divided between shards. The README states that enforcement then becomes approximate: a limit smaller than the number of shards can be exceeded in aggregate. Keep num_shards at 1 until you measure contention.

Components that are still moving

The processor has beta stability for traces. « Inverted » decisions (invert_match) are deprecated in favour of drop and not, and external storage (tail_storage) sits behind a feature gate under active development. Re-read the README with every Collector upgrade.

How to check that it works

Monitor the processor like any other service. The README lists the metrics to follow: traces evicted before a decision, trace age at removal, evaluation time and the share of traces kept.

Query file: the sampling\_trace\_dropped\_too\_early, sampling\_trace\_removal\_age and sampling\_decision\_timer\_latency metrics and the global\_count\_traces\_sampled ratio of the tail\_sampling processor

  1. The sampling_trace_dropped_too_early metric must stay at zero. If not, raise num_traces or lower decision_wait.
  2. Compare the p1 of sampling_trace_removal_age with decision_wait: if it gets close, any traffic increase causes losses.
  3. A sampling_decision_timer_latency above one second can delay decisions beyond decision_wait, according to the README.
  4. To learn which rule kept a trace, enable the processor.tailsamplingprocessor.recordpolicy gate: the tailsampling.policy attribute is then added to kept spans.

Best practices

  • Size the memory first, then the rules: a perfect rule set is useless if the buffer evicts traces.
  • Put exclusions (drop) and safety nets (errors, slowness) in place before the cost ceiling.
  • Reserve a share of the ceiling for critical services instead of one uniform percentage.
  • Turn on the decision cache as soon as long traces or late spans exist.
  • Keep your own sizing script in the repository, with its assumptions, so you can rerun it whenever traffic changes.
  • For SDK-side sampling, the Sampling page on opentelemetry.io remains the reference.

Frequently asked questions

What value should decision_wait have?

The duration of the longest trace you want to see complete, and no more. The default is 30 s. The larger the value, the more memory you need and the later the decisions arrive.

Do tail sampling rules replace sampling in the SDKs?

No, they complement it. Tail sampling only sees what the SDKs send: if an SDK samples at the head, it discards traces before the Collector can judge them. For the step-by-step setup, see the OpenTelemetry Collector on Kubernetes.

Do I need one Collector per application instance?

No. You need a second tier of Collectors that receives all traces by trace ID, as described above. The exact number depends on your throughput: measure it with the calculation in the first section.

Conclusion

Effective tail sampling rules start with a memory budget and a decision delay, then a clear composition: exclude the noise, keep errors and slow requests, cap the rest. Then watch the processor’s three metrics, and re-read the documentation at each version. Reference: the tail_sampling processor README.

span.observer is an independent community blog, not affiliated with Dynatrace LLC. Trademarks belong to their owners. The examples are fictional and were not run in a Collector: check the options in the official documentation for your version.


Originally published on span.observer. French version: https://span.observer/fr/regles-tail-sampling-collector-cout-pieges/.

Top comments (0)