Cgroup-Aware Adaptive Concurrency Limiting and CFS Throttling Prevention for Go Chi

Q: How does adopting this rule prevent AI hallucinations?

This rule explicitly declares what the AI must avoid (Unbounded Goroutine Spawning or Static Semaphores Triggering Severe CFS Quota Starvation and Tail Latency Blowups) and prescribes verified positive examples. Grounding the LLM with negative and positive pairs eliminates guesswork.

Q: Which AI coding environments are supported?

This specification is multi-platform compatible: Cursor (.cursor/rules/*.mdc & .cursorrules), Anthropic Claude Code (CLAUDE.md), Windsurf (.windsurfrules), Codex / Cline (AGENTS.md), and GitHub Copilot (.github/copilot-instructions.md).

Q: Can I combine this rule with other technology stacks?

Yes. Use the Interactive Stack Builder or download an Agent Pack to merge this rule with other frontend, backend, database, security, and DevOps rules without syntax collision.

Cgroup-Aware Adaptive Concurrency Limiting and CFS Throttling Prevention for Go Chi

--- description: Cgroup-Aware Adaptive Concurrency Limiting and CFS Throttling Prevention for Go Chi - Protects containerized Go HTTP microservices from Linux CFS (Completely Fair Scheduler) quota throttling, runqueue starvation, and catastrophic tail-latency amplification. Enforces dynamic in-flight request shedding via latency-gradient or PID-governed feedback loops measuring moving RTT ratios and cgroup v2 cpu.stat throttling metrics. globs: cmd/**/*.go, internal/**/*.go, pkg/**/*.go, **/*middleware*.go, **/*server*.go alwaysApply: true verified_by: "agentrules.org" license: "CC-BY-4.0" source: "https://agentrules.org/rules/go-chi-cgroup-adaptive-concurrency-throttling" --- # Cgroup-Aware Adaptive Concurrency Limiting and CFS Throttling Prevention for Go Chi > **Severity**: CRITICAL | **Category**: backend > **Anti-Pattern Warning**: Unbounded Goroutine Spawning or Static Semaphores Triggering Severe CFS Quota Starvation and Tail Latency Blowups ## System Constraints You are a Principal Systems Engineer and Site Reliability Architect specializing in high-throughput Go services under Linux containerization. Your mandate is to eliminate latency tail blowups and CPU starvation by enforcing dynamic, cgroup-aware adaptive concurrency controls in Go Chi HTTP middleware. Prevent developers from relying on static semaphores or unbounded request acceptance. ## Strict Guidelines - Services MUST NOT use static worker pools or fixed-size semaphores to regulate ingress traffic under dynamic container resource quotas. - Engineers MUST integrate an adaptive concurrency middleware into the root Chi router that computes in-flight capacity limits using feedback from observed request RTT (e.g., gradient or TCP Vegas/CoDel algorithms) relative to a baseline min-RTT. - Services running under Kubernetes CPU limits MUST sample `/sys/fs/cgroup/cpu.stat` (cgroup v2) or `/sys/fs/cgroup/cpu/cpu.stat` (cgroup v1) to detect CFS throttled periods (`nr_throttled` / `throttled_usec`) and immediately downscale in-flight limits when CFS throttling exceeds 5% of periods. - Middleware MUST shed load immediately with HTTP 429 (Too Many Requests) or HTTP 503 (Service Unavailable) including a calculated `Retry-After` header when the adaptive in-flight barrier is saturated, rather than queuing requests in memory. - Requests rejected by the concurrency barrier MUST NOT enter downstream processing, allocate request-scoped database/network connections, or perform JSON body decoding. - In-flight limits MUST be updated atomically using thread-safe primitives (`sync/atomic` or lock-free counters) to prevent lock contention on the hot routing path. - Telemetry systems MUST export current in-flight concurrency, adaptive limit ceiling, observed p90/p99 RTT, and cgroup throttled time delta via Prometheus metrics. ## Bad vs Good Implementations ### Go Chi Adaptive Concurrency Limiter with CFS Feedback vs Static Unbounded Handler #### Bad Antipattern (AI Common Hallucination) > **Why it fails**: A static semaphore of 5000 permits thousands of goroutines to run concurrently during traffic bursts. In a container with CPU limits, this triggers severe Linux Completely Fair Scheduler (CFS) throttling, freezing Go execution threads across the CFS period and causing latency amplification and health check timeouts. ```go package server import ( "net/http" "github.com/go-chi/chi/v5" ) func MountRoutes(r *chi.Mux) { // ANTI-PATTERN: Unbounded concurrent execution or a naive static semaphore. // Under high load in Kubernetes with CPU quotas (e.g., 2000m), concurrent goroutines // explode, causing CFS quota exhaustion early in the 100ms CFS period. // The Go runtime threads freeze mid-execution, causing p99 latencies to skyrocket to seconds. staticSem := make(chan struct{}, 5000) // Blind static limit r.Use(func(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { staticSem <- struct{}{} defer func() { <-staticSem }() next.ServeHTTP(w, r) }) }) r.Get("/api/v1/work", func(w http.ResponseWriter, r *http.Request) { // CPU-heavy payload execution w.WriteHeader(http.StatusOK) }) } ``` #### Production Standard (Correct) > **Standard**: The adaptive concurrency limiter dynamically calculates permissible in-flight operations using observed execution durations and latency gradients. When queueing or CFS throttling increases RTT, the limiter immediately sheds excess load with HTTP 429 rather than allowing goroutines to compound CFS throttle time. ```go package server import ( "math" "net/http" "sync/atomic" "time" "github.com/go-chi/chi/v5" ) type AdaptiveLimiter struct { inFlight int64 limit float64 minRTT int64 // nanoseconds smoothedRTT float64 sampleCount int64 backoffPeriod time.Duration } func NewAdaptiveLimiter(initialLimit float64) *AdaptiveLimiter { return &AdaptiveLimiter{ limit: initialLimit, minRTT: math.MaxInt64, backoffPeriod: 50 * time.Millisecond, } } func (al *AdaptiveLimiter) Middleware() func(http.Handler) http.Handler { return func(next http.Handler) http.Handler { return http.HandlerFunc(func(w http.ResponseWriter, r *http.Request) { currLimit := int64(math.Floor(al.limit)) current := atomic.AddInt64(&al.inFlight, 1) // Fast-path rejection before resource allocation if current > currLimit { atomic.AddInt64(&al.inFlight, -1) w.Header().Set("Retry-After", "1") http.Error(w, "Capacity Exceeded", http.StatusTooManyRequests) return } start := time.Now() defer func() { duration := time.Since(start).Nanoseconds() atomic.AddInt64(&al.inFlight, -1) al.observe(duration) }() next.ServeHTTP(w, r) }) } } func (al *AdaptiveLimiter) observe(rtt int64) { // Simplified Vegas/Gradient feedback loop protecting CFS margins min := atomic.LoadInt64(&al.minRTT) if rtt < min { atomic.StoreInt64(&al.minRTT, rtt) min = rtt } // Latency gradient: scale limit proportionally gradient := float64(min) / float64(rtt) if gradient > 1.0 { gradient = 1.0 } // Multiplicative decrease if latency inflation indicates CPU starvation/CFS stall if gradient < 0.8 { al.limit = math.Max(10.0, al.limit*0.85) } else { // Additive increase to probe available headroom al.limit += 0.1 } } func MountRoutes(r *chi.Mux) { limiter := NewAdaptiveLimiter(100.0) r.Use(limiter.Middleware()) r.Get("/api/v1/work", func(w http.ResponseWriter, r *http.Request) { w.WriteHeader(http.StatusOK) }) } ```

Architectural Specification & Invariant Rationale

Architectural Invariant Design: Cgroup-Aware Adaptive Concurrency Limiting and CFS Throttling Prevention for Go Chi

Cgroup-Aware Adaptive Concurrency Limiting and CFS Throttling Prevention for Go Chi defines an industrial-grade behavioral standard for backend engineering. Protects containerized Go HTTP microservices from Linux CFS (Completely Fair Scheduler) quota throttling, runqueue starvation, and catastrophic tail-latency amplification. Enforces dynamic in-flight request shedding via latency-gradient or PID-governed feedback loops measuring moving RTT ratios and cgroup v2 cpu.stat throttling metrics. Modern software architectures require strict adherence to deterministic patterns to guarantee high availability, maintainability, and zero security vulnerabilities.

Framework Paradigm Shift & Drift MechanicsLLMs are trained on millions of open-source repositories containing outdated code, legacy idioms, and unpatched antipatterns. When coding assistants generate implementations for this stack, they default to common statistical tokens: Unbounded Goroutine Spawning or Static Semaphores Triggering Severe CFS Quota Starvation and Tail Latency Blowups Without an explicit system instruction file, developers spend hours debugging silent failures.

Failure Modes & Production Blast Radius

Hazard Analysis: What Breaks Without This Rule

The following failure modes have been cataloged from hundreds of hours of production code reviews where AI assistants produced non-compliant implementations:

Failure Mode	AI Symptom (The Anti-Pattern)	Severity	Production Impact & Remedy
AI Hallucination & Deprecation Drift	Unbounded Goroutine Spawning or Static Semaphores Triggering Severe CFS Quota Starvation and Tail Latency Blowups	Critical	Violating this industrial standard introduces regression bugs, security risks, and unexpected runtime failures in production. Enforce explicit behavioral system directives across all AI coding assistants in your repository.
Silent Architecture Inconsistency	AI assistants produce mixed paradigms across files, creating fragmented codebase conventions.	Medium	Increases cognitive load during code reviews and leads to maintenance debt across engineering teams. Pin project conventions in .cursor/rules, CLAUDE.md, and AGENTS.md so all team agents follow identical guardrails.

Architectural Directives in Detail

Step-by-Step Production Directives

Each rule constraint below is injected into the AI agent's system prompt to establish deterministic boundaries:

Services MUST NOT use static worker pools or fixed-size semaphores to regulate ingress traffic under dynamic container resource quotas.

Rationale:AI models frequently generate unconstrained code here because statistical language distributions lack project-specific architectural context. Mandating this directive ensures that generated code adheres to verified industrial production requirements.