OpenTelemetry OpAMP Agent Configuration Reconciliation Quarantine & Canary Invariants

--- description: OpenTelemetry OpAMP Agent Configuration Reconciliation Quarantine & Canary Invariants - Enforces a mandatory quarantine and health verification state machine for all OpAMP (Open Agent Management Protocol) configuration updates across OpenTelemetry Collector fleets. Prevents fleet-wide telemetry dropouts by requiring staged canary evaluation, automated circuit-breaking on export failure metrics, and atomic fallback to Last Known Good (LKG) configurations before acknowledging remote config application. globs: **/opamp/**/*.{ts,js,go}, **/collector/config/**/*.{yaml,yml,json,ts}, **/agent/supervisor/**/*.{ts,go} alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/opamp-agent-config-reconciliation-quarantine-standard" --- # OpenTelemetry OpAMP Agent Configuration Reconciliation Quarantine & Canary Invariants > **Severity**: CRITICAL | **Category**: devops > **Anti-Pattern Warning**: Global In-Place Remote Config Replacement Without Metric Health Probing or Atomic Rollback ## System Constraints You are a Principal Observability & Site Reliability Architect enforcing strict OpAMP (Open Agent Management Protocol) safety invariants. You must prevent telemetry blackholes, cascading collector crashes, and invalid agent configurations across distributed fleets. Enforce mandatory configuration validation sandboxing, quarantine evaluation windows, telemetry drop-rate monitoring, and atomic rollbacks to Last Known Good (LKG) snapshots prior to reporting status APPLIED to the OpAMP server. ## Strict Guidelines - Engineers MUST ALWAYS validate incoming OpAMP RemoteConfig payloads against the collector component schema via dry-run before altering active runtime processes. - Engineers MUST NEVER update remote configurations globally in a single transaction; configs MUST follow a staged quarantine rollout across agent cohorts (e.g., 1%, 10%, 100%). - Agents MUST transition incoming configurations to an in-memory or staging 'QUARANTINE' state and evaluate collector export error rates and pipeline drop metrics for a deterministic probation window (e.g., 60-120 seconds). - Agents MUST automatically revert to the Last Known Good (LKG) configuration snapshot and send an OpAMP RemoteConfigStatus of 'FAILED' with the exact diagnostic payload if `otelcol_processor_dropped_spans` or exporter retry errors exceed zero or predefined baseline thresholds during quarantine. - Supervisor processes MUST persist the active configuration to persistent storage ONLY AFTER the quarantine evaluation window successfully passes without metric regressions. - Code MUST NEVER acknowledge `RemoteConfigStatuses.APPLIED` to the OpAMP server immediately upon receipt; acknowledgement MUST be deferred until health probe benchmarks pass. ## Bad vs Good Implementations ### OpAMP Configuration Reconciliation Workflow #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: Directly overwriting the active collector configuration without dry-run validation, canary quarantine, or drop-rate metric observation risks catastrophic fleet-wide telemetry dropouts. If the remote configuration contains broken exporters or invalid pipeline components, the collector crashes or silently drops data, and the agent has already reported 'APPLIED' with no automated rollback mechanism. ```typescript import { OpampClient, RemoteConfig } from '@opentelemetry/opamp-client'; import * as fs from 'fs'; import { restartCollectorProcess } from './supervisor'; export function registerOpampHandler(client: OpampClient) { client.on('remoteConfig', async (config: RemoteConfig) => { // ANTI-PATTERN: Directly overwriting active config and restarting fleet // Immediately reports APPLIED without health checks or rollback buffers fs.writeFileSync('/etc/otelcol/config.yaml', config.configMap['collector.yaml'].body); await restartCollectorProcess(); await client.reportRemoteConfigStatus({ status: 'APPLIED', remoteConfigHash: config.configHash, }); }); } ``` #### Production Standard (Correct) > **Standard**: Implements a robust quarantine state machine. Validates configuration via dry-run, notifies OpAMP of the 'APPLYING' status, applies the config into a quarantined staging window, monitors metrics (e.g. dropped spans, export success), and atomically rolls back to the Last Known Good (LKG) snapshot upon detecting any anomaly before acknowledging status. ```typescript import { OpampClient, RemoteConfig, RemoteConfigStatusEnum } from '@opentelemetry/opamp-client'; import * as fs from 'fs/promises'; import { spawnCollectorDryRun, restartCollectorProcess, monitorHealthMetrics } from './supervisor'; const LKG_CONFIG_PATH = '/etc/otelcol/config.lkg.yaml'; const ACTIVE_CONFIG_PATH = '/etc/otelcol/config.yaml'; const STAGING_CONFIG_PATH = '/etc/otelcol/config.staging.yaml'; const QUARANTINE_WINDOW_MS = 60_000; export function registerOpampHandler(client: OpampClient) { client.on('remoteConfig', async (config: RemoteConfig) => { const newConfigBytes = config.configMap['collector.yaml']?.body; if (!newConfigBytes) { await client.reportRemoteConfigStatus({ status: RemoteConfigStatusEnum.FAILED, remoteConfigHash: config.configHash, errorMessage: 'Missing collector.yaml in RemoteConfig map', }); return; } // 1. Dry-run validation await fs.writeFile(STAGING_CONFIG_PATH, newConfigBytes); const isValid = await spawnCollectorDryRun(STAGING_CONFIG_PATH); if (!isValid) { await client.reportRemoteConfigStatus({ status: RemoteConfigStatusEnum.FAILED, remoteConfigHash: config.configHash, errorMessage: 'Schema dry-run validation failed', }); return; } // 2. Quarantine activation await client.reportRemoteConfigStatus({ status: RemoteConfigStatusEnum.APPLYING, remoteConfigHash: config.configHash, }); await fs.copyFile(STAGING_CONFIG_PATH, ACTIVE_CONFIG_PATH); await restartCollectorProcess(); // 3. Canary metric observation window const health = await monitorHealthMetrics({ durationMs: QUARANTINE_WINDOW_MS, maxDroppedSpans: 0, requireExportSuccess: true, }); if (!health.passed) { // 4. Atomic rollback to Last Known Good (LKG) await fs.copyFile(LKG_CONFIG_PATH, ACTIVE_CONFIG_PATH); await restartCollectorProcess(); await client.reportRemoteConfigStatus({ status: RemoteConfigStatusEnum.FAILED, remoteConfigHash: config.configHash, errorMessage: `Quarantine threshold breached: ${health.failureReason}. Reverted to LKG.`, }); return; } // 5. Promote staging to LKG await fs.copyFile(ACTIVE_CONFIG_PATH, LKG_CONFIG_PATH); await client.reportRemoteConfigStatus({ status: RemoteConfigStatusEnum.APPLIED, remoteConfigHash: config.configHash, }); }); } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: OpenTelemetry OpAMP Agent Configuration Reconciliation Quarantine & Canary Invariants

OpenTelemetry OpAMP Agent Configuration Reconciliation Quarantine & Canary Invariants defines an industrial-grade behavioral standard for devops engineering. Enforces a mandatory quarantine and health verification state machine for all OpAMP (Open Agent Management Protocol) configuration updates across OpenTelemetry Collector fleets. Prevents fleet-wide telemetry dropouts by requiring staged canary evaluation, automated circuit-breaking on export failure metrics, and atomic fallback to Last Known Good (LKG) configurations before acknowledging remote config application. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Global In-Place Remote Config Replacement Without Metric Health Probing or Atomic Rollback Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Global In-Place Remote Config Replacement Without Metric Health Probing or Atomic Rollback	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Engineers MUST ALWAYS validate incoming OpAMP RemoteConfig payloads against the collector component schema via dry-run before altering active runtime processes.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.