Designing a system for one million requests per second

Start with a precise workload

Assume a sustained total of 1,000,000 requests per second at the public edge: 95% public product reads and 5% authenticated writes. Assume a mean response body of 1,000 bytes. The read responses may be shared between users; account details and writes bypass the shared cache. These assumptions describe this example, not every large application.

Define service-level objectives before selecting infrastructure. An example target is 99.9% successful responses and p99 latency below 200 milliseconds, measured at specified client locations over a stated interval and reported separately for reads and writes. A fast error does not count as a successful request. A write succeeds only after the promised durability boundary is reached.

• Specify whether one million RPS is sustained traffic, a brief peak, or traffic after a regional failure.
• Record payload sizes, endpoint mix, geographic distribution, authentication cost, and the popularity of individual keys.
• Choose acceptable data staleness and define what acknowledged writes must survive.

Translate requests into resource budgets

At a sustained million RPS, the system serves 86.4 billion requests per day. With the assumed 1,000-byte response, response bodies alone consume about 1 GB/s, or 8 Gbit/s, across the public serving fleet. Headers, TLS, incoming bodies, replication, and internal calls add more traffic. These are decimal units and aggregate totals, not the bandwidth of each server.

Concurrency is a different quantity from throughput. Little's law gives average in-flight work as arrival rate multiplied by mean time in the system, for a stable system at the same measurement boundary. At one million RPS and a mean response time of 100 milliseconds, that is about 100,000 requests in flight. Do not substitute p99 latency for the mean or confuse requests with TCP connections.

Daily requests = 1,000,000 × 86,400
               = 86,400,000,000

Response bytes = 1,000,000 × 1,000 bytes/second
               = 1 GB/second ≈ 8 Gbit/second

Mean in flight = 1,000,000/second × 0.100 seconds
               = 100,000 requests

Use caching to remove repeated work

Suppose the edge cache serves 90% of the 950,000 public reads each second. It handles 855,000 RPS, leaving 95,000 read misses plus 50,000 writes: 145,000 RPS at the origin. The hit ratio applies to eligible reads, not to all requests. One million edge requests therefore does not mean one million application executions.

Configure cache eligibility explicitly. For example, Cloudflare does not cache HTML or JSON by default. Validate the cache key, response headers, query parameters, and any cookie or authorization behavior before caching an API response. Private data must not enter a public shared cache. Freshness requirements determine the lifetime; cache-hit targets cannot override correctness.

Plan for simultaneous misses. Request collapsing can combine concurrent fetches for the same key; Cloudflare documents this behavior within a single data center. At the application layer, consider bounded refresh work and staggered expiry. After an edge-cache purge, origin demand could rise from 145,000 toward 1,000,000 RPS, about 6.9 times the warm-cache estimate. Measure that failure case explicitly.

Reference: Cloudflare: default cache behavior and request collapsing ↗

Distribute traffic and isolate failures

Use a global routing layer to choose a healthy region with available capacity, then regional load balancers to distribute origin requests across application replicas. Within a region, place replicas across availability zones. A region contains a serving fleet and its data dependencies; adding regions requires a deliberate replication and write-ownership strategy.

Keep application replicas stateless so another healthy replica can handle the next request. For a Next.js product, static pages and assets can use the CDN while dynamic APIs run on independently sized compute. In a Node.js service, move CPU-heavy work off the request event loop. Database pools, authentication dependencies, and load-balancer limits still need their own budgets.

Reference: Google SRE: load balancing at the frontend ↗

Size replicas from a representative benchmark

Suppose a future benchmark measures 4,000 RPS per application instance while meeting the latency and error targets for the origin workload. This is an illustrative input, not a measured result. Choosing 70% of that rate as the operating budget gives 2,800 RPS per instance. The baseline is then the ceiling of 145,000 divided by 2,800: 52 instances.

For a simplified single-region calculation, spread the entire origin load across three equally provisioned zones and require capacity after losing one zone. Each of the two surviving zones needs 26 instances, so provision 78 in total. This only sizes application compute under the assumed cache behavior. Uneven traffic, database failover, rolling deployments, or a colder cache may demand more capacity.

const edgeRps = 1_000_000;
const readShare = 0.95;
const readCacheHit = 0.90;
const originRps = Math.round(
  edgeRps * (1 - readShare * readCacheHit)
);

// Hypothetical benchmark and chosen headroom.
const benchmarkRpsPerInstance = 4_000;
const operatingFraction = 0.70;
const budget = benchmarkRpsPerInstance * operatingFraction;

const zones = 3;
const zonesToSurvive = zones - 1;
const baseline = Math.ceil(originRps / budget);
const perZone = Math.ceil(originRps / (zonesToSurvive * budget));

console.log({ originRps, baseline, perZone,
  totalWithZoneReserve: perZone * zones });
// 145000, 52, 26, 78

Find the database bottleneck before adding replicas

Assume an application cache answers 90% of the remaining 95,000 reads. That leaves 9,500 database read requests and 50,000 write requests per second. If each read issues two queries and each write issues one database operation, the first-pass budget is 69,000 operations per second. Index updates, transactions, replication, and outbox records increase the real storage work.

Measure query plans, lock waits, connection-pool queues, and the hottest keys. Partitioning data across independent database shards can distribute work, but the shard key must match access patterns. A popular tenant or product can saturate one partition while others remain idle. DynamoDB's partition-key guidance illustrates why uniform activity, item size, and per-partition limits matter even with a managed database.

Choose read consistency deliberately. A lagging read replica may be acceptable for a product description but unsuitable for confirming a just-completed operation. Keep strongly related writes together when possible; cross-shard transactions and global uniqueness add coordination. Adding API replicas without limiting their combined database connections can make the bottleneck worse.

Reference: AWS DynamoDB: designing partition keys effectively ↗

Move optional work behind a durable queue

Keep only the work required by the response contract on the synchronous path. Search indexing, email, and analytics can often happen later. Commit the business write and an outbox record together, then publish committed events to a durable queue. Consumers need stable operation IDs and idempotent effects because delivery can repeat.

A queue absorbs a temporary mismatch; it cannot provide unlimited processing capacity. If 50,000 jobs arrive each second and workers finish 45,000, the backlog grows by 300,000 jobs every minute. Track oldest-job age as well as queue depth. Limit admission or add measured worker capacity before storage and completion deadlines are exhausted. Report accepted requests and completed business operations as separate outcomes.

Reference: AWS: transactional outbox pattern ↗

Preserve useful capacity under overload

Limit in-flight requests and waiting work at each service boundary. Enforce tenant quotas before expensive work, use downstream deadlines that fit within the caller's deadline, and stop unnecessary work after cancellation. When capacity is exhausted, reject excess requests quickly or return a permitted degraded response. An unbounded queue converts overload into rising latency and memory pressure.

Retry only transient failures when repeating the operation is safe, with bounded attempts, backoff, jitter, and an overall retry budget. Coordinate retry ownership: three attempts at each of three nested layers can generate 27 attempts at the deepest dependency. During broad overload, another attempt may only consume capacity needed for recovery.

Autoscaling takes time to observe load, start instances, and warm dependencies. Keep reserve capacity and test the delay. A successful overload policy maintains useful throughput while shedding excess traffic; it does not count rejected requests as successfully handling one million RPS.

Reference: Google SRE: handling overload ↗

Prove capacity with realistic arrival-rate tests

Use a controlled environment with production-like data sizes, key popularity, endpoint mix, and dependencies. Increase offered load in stages, inspect the limiting resource, and repeat after fixing it. A fast endpoint returning a constant string says little about a product API with authentication and durable writes.

For a fixed-RPS goal, an open arrival-rate model schedules work independently of response completion. In a closed model, slow responses can reduce the traffic generated and conceal the intended stress. Grafana k6 documents this distinction. Arrival-rate executors schedule iterations, so account for how many requests each iteration sends. Monitor dropped iterations and load-generator CPU and networking; requesting a target rate does not prove it was delivered.

• Compare scheduled traffic, requests actually sent, edge receipts, origin traffic, and successful responses within the latency target.
• Use enough distributed generators to sustain the offered load, with known geographic locations and synchronized measurement windows.
• Test warm caches, cold caches, a hot key, a zone failure, a slow database, retries, and a rolling deployment.
• Run long enough to expose leaks, compaction, queue growth, and recovery. Verify stored results and duplicate handling as well as HTTP status codes.

Reference: Grafana k6: open and closed workload models ↗

Make observability and cost part of the design

Measure request rate, errors, latency distributions, saturation, cache misses, database waits, and queue age at their respective boundaries. Keep metric labels bounded: endpoint templates are useful; a unique user ID or request ID per label creates a growing number of time series. Use appropriately sampled traces and structured logs to investigate individual requests.

At one million RPS, writing one 1,000-byte log per request produces 86.4 TB of raw logs per day before replication or indexing. The same arithmetic applies to the assumed response bodies across the edge. Budget storage retention, network transfer, spare compute, and database replication alongside the main application fleet.

Build toward the target in measured steps: establish a baseline, remove redundant work, improve the limiting query or service, then distribute the bottleneck that remains. Each change should improve successful throughput at the required latency without weakening correctness or recovery. The architecture earns its capacity claim through those measurements.

Reference: Prometheus: instrumentation and label cardinality ↗