Description
Performs sampling of traces per a given rate using the Trace ID as the unique identifier. If a trace is rate selected, its spans are output. Intail mode, a span that arrives after the decision expires can be held or sampled again; see How tail sampling closes a trace.
This is an experimental feature that is still in development. For access to this feature, reach out to your Customer Support Manager or support@mezmo.com.
Use
Two well-known optimization techniques for traces are Deterministic Sampling (Head sampling) and more advanced Tail-based sampling. While head-based sampling will treat all traces equally and sample randomly, Tail-based sampling enables you to sample at different rates for traces you consider to have different levels of importance, based on attributes that traces contain, for example prioritizing traces with errors, latency, or traces that exceed certain performance thresholds. This is implemented in this Processor through conditional statements to define the trace.This Processor supports sampling of OTel traces only.
Configuration
By default, the Trace ID and Parent Span ID fields point to
.record.traceId and .record.parentSpanId. These paths match the structure produced by the combined OpenTelemetry Source, which nests traceId and parentSpanId under a .record property, so the Processor works with that Source without extra setup. If your traces store these values elsewhere, set each field to the path where the value lives.How tail sampling closes a trace
Intail mode, the Processor decides a trace when the head span of the trace arrives. The head span is the span that has no Parent Span ID: the field is missing, null, an empty string, or not a string. The Processor does not wait for a timer, a maximum trace duration, or a span count before it decides.
The Processor handles a trace in this order:
- A span that has a Parent Span ID and arrives before the head span is held. The Processor does not output it yet.
- When the head span arrives, the Processor checks it against the conditionals. If there is more than one conditional, the first one that matches is used. Only the head span is checked, so the conditional must use fields that the head span carries.
- The Processor applies the Rate of the matching conditional. A Rate of
1keeps every matching trace. A Rate of10keeps the first matching trace and then 1 of every 10 matching traces. A trace that matches no conditional is not kept. - If the trace is kept, the Processor outputs the head span and all the held spans together. If the trace is not kept, the Processor discards the held spans.
Time limit on a decision
The Processor keeps the decision, and any held spans, for at least 5 minutes. The time limit is not an option of this Processor. The storage layer removes expired records on a best-effort basis, so a record can outlast the limit. After the limit passes, the Processor no longer knows the trace:- A late span that has a Parent Span ID is held again. The Processor does not output it unless a head span for the same trace arrives.
- A late span that has no Parent Span ID is treated as a new head span. The Processor makes a new decision for the trace, and applies the Rate again.
- A trace whose head span never arrives is never output. Its held spans expire.
- A span without a trace ID is dropped, and the Processor records a warning.
- An event whose message is not an object is dropped, and the Processor records a warning.

