OpenTelemetry Collector High-Watermark Filelog Checkpoint Persistence Invariant

--- description: OpenTelemetry Collector High-Watermark Filelog Checkpoint Persistence Invariant - Mandates durable, disk-persisted read offsets via the file_storage extension and persistent volume mounts for the OpenTelemetry Collector filelog receiver. Prevents duplicate log ingestion floods and telemetry drops caused by ephemeral container restarts resetting in-memory high-watermarks during log rotation. Enforces explicit storage extension linkage and persistent node-local filesystem backing across all Collector deployments. globs: **/collector-*.yaml, **/collector-*.yml, **/otel-collector*.yaml, **/otel-collector*.yml, **/configmap*.yaml, **/daemonset*.yaml alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/otel-filelog-checkpoint-persistence-invariant" --- # OpenTelemetry Collector High-Watermark Filelog Checkpoint Persistence Invariant > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Volatile In-Memory Filelog Checkpointing and Ephemeral Pod Storage Leading to Duplicate Telemetry Storms and Data Loss ## System Constraints You are a Principal Software Architect enforcing production data durability and telemetry invariants. You must prevent telemetry duplication spikes and data loss in OpenTelemetry Collector deployments by enforcing synchronous, disk-backed read offset checkpoints (file_storage extension) mounted to durable non-ephemeral storage across all filelog receiver configurations. ## Strict Guidelines - Engineers MUST ALWAYS configure the 'file_storage' extension under 'extensions' in OpenTelemetry Collector configuration when using the 'filelog' receiver. - Engineers MUST ALWAYS link 'receivers.filelog.storage: file_storage' explicitly in collector pipelines to prevent falling back to default volatile in-memory checkpoints. - Services MUST mount the 'file_storage' directory to a durable persistent volume or host-path mount; engineers MUST NEVER use ephemeral 'emptyDir' storage for log offset directories. - Engineers MUST configure 'compaction.on_rebound: true' and 'create_directory: true' in 'file_storage' to prevent disk exhaustion from stale file fingerprint inodes. - Components NEVER assume log offset persistence across pod lifecycles unless both the OpenTelemetry pipeline config and Kubernetes VolumeMounts validate persistent storage linkage. ## Bad vs Good Implementations ### Ephemeral In-Memory Filelog Receiver Configuration vs Durable Persistent Checkpointing #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Omitting the 'storage' parameter causes the filelog receiver to retain file read high-watermarks in volatile memory. During a pod restart, node drain, or OOM kill, the collector loses track of consumed log lines and either re-emits millions of duplicate logs (start_at: beginning) or permanently drops logs written while down (start_at: end). ```yaml # Anti-Pattern: In-memory checkpointing with ephemeral container storage apiVersion: v1 kind: ConfigMap metadata: name: otel-collector-config data: collector.yaml: | receivers: filelog: include: [/var/log/pods/*/*/*.log] # FATAL: Missing 'storage' attribute defaults to in-memory checkpoints. # Container restarts re-read files from beginning or drop rotated files. start_at: end processors: batch: exporters: otlp: endpoint: tempo-distributor:4317 service: pipelines: logs: receivers: [filelog] processors: [batch] exporters: [otlp] ``` #### Production Standard (Correct) > **Standard**: The 'file_storage' extension is explicitly defined, added to 'service.extensions', and bound to 'receivers.filelog.storage'. Combined with a checksum-based fingerprint and a mounted persistent directory, read offsets survive pod crashes, node restarts, and log rotation without duplicates or silent telemetry loss. ```yaml # Production Invariant: Durable file_storage extension with host-persisted checkpoint path apiVersion: v1 kind: ConfigMap metadata: name: otel-collector-config data: collector.yaml: | extensions: file_storage: directory: /var/lib/otelcol/checkpoints create_directory: true compaction: on_rebound: true directory: /var/lib/otelcol/checkpoints receivers: filelog: include: [/var/log/pods/*/*/*.log] exclude: [/var/log/pods/*/*/*.gz] storage: file_storage retry_on_failure: enabled: true poll_interval: 200ms fingerprint: strategy: checksum size: 1024 processors: batch: send_batch_size: 8192 timeout: 1s exporters: otlp: endpoint: tempo-distributor:4317 service: extensions: [file_storage] pipelines: logs: receivers: [filelog] processors: [batch] exporters: [otlp] ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry Collector High-Watermark Filelog Checkpoint Persistence Invariant

OpenTelemetry Collector High-Watermark Filelog Checkpoint Persistence Invariant defines an industrial-grade behavioral standard for devops engineering. Mandates durable, disk-persisted read offsets via the file_storage extension and persistent volume mounts for the OpenTelemetry Collector filelog receiver. Prevents duplicate log ingestion floods and telemetry drops caused by ephemeral container restarts resetting in-memory high-watermarks during log rotation. Enforces explicit storage extension linkage and persistent node-local filesystem backing across all Collector deployments. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Volatile In-Memory Filelog Checkpointing and Ephemeral Pod Storage Leading to Duplicate Telemetry Storms and Data Loss Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Volatile In-Memory Filelog Checkpointing and Ephemeral Pod Storage Leading to Duplicate Telemetry Storms and Data Loss	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS configure the 'file_storage' extension under 'extensions' in OpenTelemetry Collector configuration when using the 'filelog' receiver.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.