Continuous eBPF Wall-Clock Profiling to OTel Span Correlation Standard

--- description: Continuous eBPF Wall-Clock Profiling to OTel Span Correlation Standard - Enforces deterministic injection of active OpenTelemetry trace and span context into kernel-readable user-space memory locations or thread naming registries. Guarantees continuous eBPF wall-clock profilers can sample on-CPU and off-CPU stack traces correlated directly to OpenTelemetry span IDs without sampling bias or kernel tracing loss. globs: src/**/*.ts, src/**/*.go, src/**/*.rs, k8s/**/*.yaml, deploy/**/*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/ebpf-otel-span-profiling-correlation-standard" --- # Continuous eBPF Wall-Clock Profiling to OTel Span Correlation Standard > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Decoupled tracing and continuous profiling resulting in uncorrelatable tail-latency root causes ## System Constraints You are a Principal Observability & Systems Architect enforcing continuous eBPF wall-clock profiling correlation with OpenTelemetry spans. Your mandate is to eliminate blind spots in microservice tail-latency by ensuring that runtime threads and asynchronous execution contexts continuously expose their W3C TraceContext (Trace ID and Span ID) in formats accessible to user-space eBPF probes (e.g., thread names, designated TLS/uprobe scratchpads, or pprof-compatible labels), while preventing high overhead, frame-pointer truncation, and off-CPU sampling blind spots. ## Strict Guidelines - Services MUST ensure binaries and container base images are compiled with frame pointers preserved (`-fno-omit-frame-pointer`) and debug symbols available for reliable eBPF stack unwinding without heavy DWARF parsing in kernel space. - Engineers MUST set thread-local correlation tags or pprof labels containing the active W3C `trace_id` and `span_id` whenever entering a span boundary or dispatching async execution. - Engineers MUST NOT depend solely on on-CPU profiling; continuous profiling agents MUST capture both on-CPU instructions and off-CPU scheduler wait states (lock contention, epoll/IO wait) correlated to span duration. - Services MUST register consistent thread naming conventions (e.g., `prctl(PR_SET_NAME)` or OS-level thread descriptions) truncated to fit within the Linux 16-character `comm` limit or expose a dedicated shared-memory/uprobe target when the OS thread pool is multiplexed. - Profilers MUST export profiles in OTel Profiling Data Model format or pprof format annotated with exact `span_id` and `trace_id` attributes matching the active trace telemetry pipeline. ## Bad vs Good Implementations ### Correlating Go goroutines to eBPF pprof labels and trace contexts #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: The goroutine executes detached from the pprof labeling context. Continuous eBPF profilers sampling sched_switch cannot associate off-CPU scheduling latency or on-CPU hotspots with the OpenTelemetry Span ID, making latency anomalies uncorrelatable. ```go // BAD: Spawns asynchronous worker without propagating trace context to runtime profiler labels. // eBPF continuous profilers cannot correlate off-CPU lock waits to the originating trace. func HandleRequest(ctx context.Context, job Job) { tr := otel.Tracer("job-processor") ctx, span := tr.Start(ctx, "ProcessJob") defer span.End() go func() { // Off-CPU wait on mutex and I/O will appear under anonymous thread/goroutine context executeTask(job) }() } ``` #### Production Standard (Correct) > **Standard**: Explicitly invokes runtime/pprof.Do to associate the active Span ID and Trace ID with the goroutine's label set. eBPF profilers inspecting goroutine labels or integration agents can immediately aggregate wall-clock execution samples under the discrete OpenTelemetry trace context. ```go // GOOD: Binds the active OpenTelemetry TraceID and SpanID directly into runtime/pprof labels // readable by continuous eBPF profilers and downsampled sched_switch monitors. func HandleRequest(ctx context.Context, job Job) { tr := otel.Tracer("job-processor") ctx, span := tr.Start(ctx, "ProcessJob") defer span.End() spanCtx := span.SpanContext() labels := pprof.Labels( "trace_id", spanCtx.TraceID().String(), "span_id", spanCtx.SpanID().String(), ) // Propagate context and runtime labels to child goroutines go func(jobCtx context.Context) { pprof.Do(jobCtx, labels, func(c context.Context) { executeTask(c, job) }) }(ctx) } ``` ### Preserving frame pointers and configuring eBPF continuous profiler in Kubernetes #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Without frame pointers, standard eBPF unwinding fails or falls back to expensive DWARF lookups. Missing profiler correlation annotations prevents daemonsets from scoping profile targets. ```yaml # BAD: Omits frame pointer preservation and continuous profiling annotations apiVersion: apps/v1 kind: Deployment metadata: name: payment-service spec: template: spec: containers: - name: payment image: payment-service:latest resources: limits: cpu: "2" ``` #### Production Standard (Correct) > **Standard**: Applies eBPF discovery annotations and sets environment flags to guarantee clean frame pointers and JIT symbol maps for user-space to kernel-space call-stack correlation. ```yaml # GOOD: Enforces frame pointers via build flags and instruments container for eBPF OTel correlation apiVersion: apps/v1 kind: Deployment metadata: name: payment-service annotations: profiling.opentelemetry.io/enabled: "true" profiling.opentelemetry.io/sample-rate-hz: "99" profiling.opentelemetry.io/trace-context-mode: "pprof-labels" spec: template: metadata: labels: app.kubernetes.io/name: payment-service spec: containers: - name: payment image: payment-service:latest env: - name: NODE_OPTIONS value: "--perf-basic-prof --interpreted-frames-native-stack" - name: CGO_CFLAGS value: "-fno-omit-frame-pointer -mno-omit-leaf-frame-pointer" resources: limits: cpu: "2" memory: "1Gi" ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Continuous eBPF Wall-Clock Profiling to OTel Span Correlation Standard

Continuous eBPF Wall-Clock Profiling to OTel Span Correlation Standard defines an industrial-grade behavioral standard for devops engineering. Enforces deterministic injection of active OpenTelemetry trace and span context into kernel-readable user-space memory locations or thread naming registries. Guarantees continuous eBPF wall-clock profilers can sample on-CPU and off-CPU stack traces correlated directly to OpenTelemetry span IDs without sampling bias or kernel tracing loss. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Decoupled tracing and continuous profiling resulting in uncorrelatable tail-latency root causes Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Decoupled tracing and continuous profiling resulting in uncorrelatable tail-latency root causes	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Services MUST ensure binaries and container base images are compiled with frame pointers preserved (`-fno-omit-frame-pointer`) and debug symbols available for reliable eBPF stack unwinding without heavy DWARF parsing in kernel space.

Services MUST ensure binaries and container base images are compiled with frame pointers preserved (`-fno-omit-frame-pointer`) and debug symbols available for reliable eBPF stack unwinding without heavy DWARF parsing in kernel space.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.