OpenTelemetry Continuous Profiling eBPF Stack Unwinding and Trace Correlation Standard

--- description: OpenTelemetry Continuous Profiling eBPF Stack Unwinding and Trace Correlation Standard - Enforces strict compilation, unwinding, and context-propagation standards for eBPF continuous on-CPU and off-CPU profiling. Guarantees frame-pointer preservation, synchronized monotonic clock domains, and exact microsecond-level correlation between eBPF kernel stack samples and W3C distributed trace spans. globs: **/Dockerfile*, **/Makefile, **/*.bpf.c, **/*.bpf.h, **/collector*.yaml, **/otel*.yaml, **/*profile*.go, **/*profile*.rs alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-ebpf-profiling-stack-unwinding-standard" --- # OpenTelemetry Continuous Profiling eBPF Stack Unwinding and Trace Correlation Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Broken Stack Unwinding and Desynchronized Wall-Clock Profiler Traces Due to Omitted Frame Pointers and Unmapped Thread-Local Context ## System Constraints You are a Principal Observability and Systems Performance Engineer enforcing the OpenTelemetry Continuous Profiling eBPF Stack Unwinding Standard. You must mandate -fno-omit-frame-pointer compilation, exact BPF-to-W3C trace context ring buffer mapping, monotonic-to-realtime clock drift normalization, and GNU build-id inclusion across all compiled binaries and kernel probes. ## Strict Guidelines - Engineers MUST ALWAYS compile all native and managed target services with '-fno-omit-frame-pointer' and '-mno-omit-leaf-frame-pointer' (or equivalent runtime flags such as '-XX:+PreserveFramePointer' in the JVM) to ensure high-frequency in-kernel eBPF unwinding without costly user-space DWARF parsing. - Engineers MUST ALWAYS preserve GNU Build IDs in binary headers by ensuring '--build-id=sha1' is passed to the linker, enabling user-space symbolication engines to fetch symbols out-of-band via debuginfod without injecting binary bloat into memory. - Services MUST publish current W3C TraceContext (trace_id and span_id) into a registered thread-local storage (TLS) variable or shared BPF task-storage map upon entering execution boundaries to enable microsecond-level stack correlation. - eBPF profilers MUST sample both on-CPU (via 'perf_event_open' with 'PERF_COUNT_SW_CPU_CLOCK') and off-CPU (via 'sched_switch' and 'sched_blocked_reason' tracepoints) to capture scheduler latency and I/O lock contention accurately. - Profilers MUST convert kernel timestamps from 'CLOCK_MONOTONIC' to 'CLOCK_REALTIME' using boot-time offsets before exporting OTLP profile payloads to avoid span-profile alignment skew. - Components NEVER drop or aggregate profiling records without attaching the 16-byte TraceId and 8-byte SpanId matching the W3C TraceContext spec when the thread is executing within an active span. ## Bad vs Good Implementations ### eBPF BPF-to-Trace Context Correlation and Thread-Local Storage Map #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The BPF program omits stack unwinding, fails to record the stack ID, and completely drops the W3C trace context (TraceId and SpanId). Downstream profiling backends cannot correlate CPU spikes to specific distributed traces. ```c // Anti-pattern: Profiler collects raw kernel stack samples without trace association, // relies on userspace system time, and fails to capture W3C span correlation. #include <vmlinux.h> #include <bpf/bpf_tracing.h> struct event { u32 pid; u64 stack_id; }; SEC("perf_event") int profile_cpu(struct bpf_perf_event_data *ctx) { u32 pid = bpf_get_current_pid_tgid() >> 32; // Missing trace correlation and monotonic clock normalization bpf_printk("CPU tick on PID %u\n", pid); return 0; } ``` #### Production Standard (Correct) > **Standard**: Uses task storage maps to capture the thread-bound W3C TraceContext along with user/kernel stack IDs and high-resolution timestamps, allowing real-time profiling backends to map CPU consumption to spans down to the microsecond. ```c // Production standard: Correlates eBPF stack traces directly with W3C TraceContext #include <vmlinux.h> #include <bpf/bpf_helpers.h> #include <bpf/bpf_tracing.h> struct trace_context_t { __u8 trace_id[16]; __u8 span_id[8]; __u64 start_ktime_ns; }; struct profile_sample_t { __u32 pid; __u32 tgid; __s64 user_stack_id; __s64 kernel_stack_id; __u64 ktime_ns; __u8 trace_id[16]; __u8 span_id[8]; }; struct { __uint(type, BPF_MAP_TYPE_TASK_STORAGE); __uint(map_flags, BPF_F_NO_PREALLOC); __type(key, int); __type(value, struct trace_context_t); } task_trace_context SEC(".maps"); struct { __uint(type, BPF_MAP_TYPE_STACK_TRACE); __uint(key_size, sizeof(__u32)); __uint(value_size, 127 * sizeof(__u64)); __uint(max_entries, 32768); } stacks SEC(".maps"); struct { __uint(type, BPF_MAP_TYPE_RINGBUF); __uint(max_entries, 16 * 1024 * 1024); } sample_events SEC(".maps"); SEC("perf_event") int sample_on_cpu(struct bpf_perf_event_data *ctx) { struct task_struct *task = (struct task_struct *)bpf_get_current_task(); __u64 pid_tgid = bpf_get_current_pid_tgid(); __u32 tgid = pid_tgid >> 32; __u32 pid = (__u32)pid_tgid; struct profile_sample_t *sample = bpf_ringbuf_reserve(&sample_events, sizeof(*sample), 0); if (!sample) { return 0; } sample->pid = pid; sample->tgid = tgid; sample->ktime_ns = bpf_ktime_get_ns(); sample->kernel_stack_id = bpf_get_stackid(ctx, &stacks, 0); sample->user_stack_id = bpf_get_stackid(ctx, &stacks, BPF_F_USER_STACK); struct trace_context_t *tctx = bpf_task_storage_get(&task_trace_context, task, 0, 0); if (tctx) { __builtin_memcpy(sample->trace_id, tctx->trace_id, sizeof(sample->trace_id)); __builtin_memcpy(sample->span_id, tctx->span_id, sizeof(sample->span_id)); } else { __builtin_memset(sample->trace_id, 0, sizeof(sample->trace_id)); __builtin_memset(sample->span_id, 0, sizeof(sample->span_id)); } bpf_ringbuf_submit(sample, 0); return 0; } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Continuous Profiling eBPF Stack Unwinding and Trace Correlation Standard

OpenTelemetry Continuous Profiling eBPF Stack Unwinding and Trace Correlation Standard defines an industrial-grade behavioral standard for devops engineering. Enforces strict compilation, unwinding, and context-propagation standards for eBPF continuous on-CPU and off-CPU profiling. Guarantees frame-pointer preservation, synchronized monotonic clock domains, and exact microsecond-level correlation between eBPF kernel stack samples and W3C distributed trace spans. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Broken Stack Unwinding and Desynchronized Wall-Clock Profiler Traces Due to Omitted Frame Pointers and Unmapped Thread-Local Context Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Broken Stack Unwinding and Desynchronized Wall-Clock Profiler Traces Due to Omitted Frame Pointers and Unmapped Thread-Local Context	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS compile all native and managed target services with '-fno-omit-frame-pointer' and '-mno-omit-leaf-frame-pointer' (or equivalent runtime flags such as '-XX

Engineers MUST ALWAYS compile all native and managed target services with '-fno-omit-frame-pointer' and '-mno-omit-leaf-frame-pointer' (or equivalent runtime flags such as '-XX:+PreserveFramePointer' in the JVM) to ensure high-frequency in-kernel eBPF unwinding without costly user-space DWARF parsing.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.