Load Shedding and Request Prioritization: Keeping Critical Flows Alive During Outages
Introduction
Your payment processing service is drowning. A bot attack floods your API with 50,000 requests per second—ten times your normal traffic. Meanwhile, legitimate users trying to complete checkouts are timing out. Your database connections are exhausted, CPU is pegged at 100%, and response times have degraded from 50ms to 8 seconds. The traditional approach—accepting all requests and letting everything fail slowly—creates cascading failures across dependent services. Load shedding is the counterintuitive solution: deliberately reject low-priority requests so critical operations survive.
The Mechanism Behind Load Shedding
Load shedding operates on a simple principle: when system capacity is exceeded, reject requests proactively rather than accepting everything and failing slowly. The system measures current load (CPU, memory, queue depth, latency) against configured thresholds. When thresholds are breached, the admission controller starts rejecting requests based on priority classification.
Priority classification happens at the edge before expensive operations begin. Each request gets tagged with a priority level—typically P0 (critical), P1 (important), P2 (normal), P3 (background). The classification uses multiple signals: authentication status (logged-in users rank higher), request type (checkout vs browsing), user tier (paid vs free), endpoint (critical APIs vs analytics), and historical behavior (new users vs established customers).
The admission controller maintains a acceptance probability for each priority level. Under normal load, all priorities are accepted. As load increases, P3 requests are rejected first, then P2, then P1. P0 requests are never rejected unless the system is completely overwhelmed. The rejection happens immediately with a 503 Service Unavailable response, consuming minimal resources—just enough to classify the request and return the rejection.
