API Gateway Best Practices: Taming One Failure Domain

Every microservices architecture eventually grows an API gateway. It starts as sensible infrastructure: one place to terminate TLS, one place to check tokens, one place to decide which service handles a routing request. Then it accumulates. Authentication and authorization move in, then rate limiting and throttling, then request and response transformation. Six months later the gateway is the most business-critical process in the fleet.

An API gateway is not just a router. It is a shared failure domain that four unrelated concerns were quietly merged into, and when one degrades they all degrade together. Most api gateway best practices exist for exactly this reason: to stop the gateway from becoming an api gateway single point of failure for everything behind it. This post covers the failure modes and the api gateway design patterns that keep it boring.

What an API gateway actually does

At its narrowest, a gateway is a reverse proxy with a routing table. It matches a path or host against a rule and forwards to an upstream service. If that were all it did, it would be about as risky as load balancing.

It never stays narrow. In practice it owns four responsibilities, added in roughly this order.

• Routing. Path-based, host-based, header-based, and eventually weighted routing for canary deploys. The table starts with ten rules and ends with four hundred, most added by teams who do not own the gateway.
• Authentication and authorization. Validating a JWT, resolving api keys to a tenant, attaching scopes so downstream services do not each implement token validation differently.
• Throttling. Per-tenant limits, per-endpoint quotas, and abuse protection based on the number of requests a caller makes. This needs state, which is the root of most gateway trouble.
• Transformation. Rewriting paths, injecting headers, converting REST to gRPC, aggregating an api call to two upstreams into one response. This is where CPU cost hides.

Each is defensible alone. The problem is that they share one process, one thread pool, one health check, and one deploy cadence.

Why the gateway becomes a single point of failure

A single point of failure is usually defined as a component with no redundancy. That is too generous to gateways, which almost always run many replicas. Losing one replica is a non-event.

The real problem is correlated failure. Redundancy protects against independent faults, and gateway faults are rarely independent. Every replica runs the same config, talks to the same identity provider, and shares the same rate limit store. A bad route reload takes down all of them within seconds. Four failure modes are worth knowing by name.

Failure mode 1: the synchronous auth dependency

The gateway validates every single request against an identity service, which now sees api traffic equal to your entire ingress volume on the critical path.

When that service slows from 5ms to 500ms, the gateway does not fail. It does something worse: it holds connections open. Each in-flight request occupies a worker slot plus a socket, and slots are finite. Within a minute the gateway is saturated with auth waits, and endpoints needing no access control at all start queueing behind them. A slow dependency on 10% of traffic becomes a total outage through shared resource exhaustion.

The fix is not a faster identity service. Verify JWT signatures locally against a cached public key set, cache introspection results with a short TTL, and set a timeout aggressive enough that a hung dependency releases the slot.

Failure mode 2: throttling state

This is where api gateway rate limiting stops being a checkbox and becomes an availability decision.

Rate limiting needs a counter shared across replicas, otherwise a limit of 1000 per minute becomes 1000 per replica. So teams put counters in Redis, and now every request makes a network round trip.

Two things go wrong. That Redis instance becomes a second synchronous dependency with the same saturation dynamics as auth. And when it is unreachable, the gateway must choose: allow or deny. Allowing means limits silently vanish during the incident where you need them most. Denying means a Redis blip becomes a full outage.

The middle ground is local-first limiting. Each replica enforces its share from an in-memory token bucket and reconciles asynchronously. You give up precision and gain the property that a dead Redis degrades accuracy rather than availability.

Failure mode 3: transformation CPU

Routing is cheap. Parsing a 2MB JSON body, rewriting fields, and re-serializing is not. Aggregation is worse, since the gateway buffers multiple upstream responses before emitting anything.

Transformation cost is not uniform across endpoints, which is what makes it dangerous. One heavy transform on a payload that grew over time can consume a disproportionate share of CPU shared by multiple services, so one team's schema change degrades latency for everyone else. Memory behaves the same way: buffering full bodies means memory scales with concurrency times payload size.

Failure mode 4: config reload

The routing table is data, not code, which is why teams change it casually. A malformed route, an overlapping prefix, or a regex that backtracks catastrophically propagates to every replica in seconds, because the config channel is built for fast propagation.

This is the failure mode that most often causes a full outage and the one least covered by your deploy safety net. Code gets CI, review, and staged rollout. Config often gets an API call.

Sizing the blast radius

Be concrete about what an outage costs. Suppose the gateway fronts 40 services at 50,000 requests per second and holds 10,000 concurrent connections. If identity latency jumps to 500ms, you need 25,000 concurrent slots to keep up and you have 10,000. The queue grows without bound, clients time out, and retries push effective load higher. The system does not degrade gracefully, it collapses across all 40 services at once.

That arithmetic is the argument for every pattern below, and being able to compute it puts you ahead of most candidates. Our post on back-of-the-envelope estimation works through the same math on other components of distributed systems.

API gateway best practices that keep it boring

Separate the data plane from the control plane

The most consequential choice in any api gateway architecture is this split. The data plane forwards requests; the control plane distributes config, collects telemetry, and manages certificates. Keep them in separate processes with separate failure domains, and let the data plane run indefinitely on its last known good config.

Envoy popularized this, which is why an Envoy data plane keeps serving when its control plane is entirely down (see the Envoy architecture overview). If your gateway cannot serve without its control plane, you have two single points of failure wearing one name.

Decide fail-open versus fail-closed per concern

Teams tend to pick one policy for the whole gateway. That is wrong, because the concerns have opposite risk profiles.

Authentication should fail closed: serving unverified traffic is a security incident, which is worse than an availability incident. Rate limiting should usually fail open, since degraded limits beat rejecting legitimate traffic, with abuse-facing endpoints like login as the exception. Transformation should fail closed for the affected route only. Write these security measures down per concern, because nobody derives them mid-incident.

Enforce rate limits locally, reconcile asynchronously

Give each replica a local token bucket sized to its share of the global budget, and publish consumption to the shared store on an interval rather than per request. A replica that cannot reach the store keeps enforcing its local share.

import time
from dataclasses import dataclass, field


@dataclass
class TokenBucket:
    """Local, network-free rate limiter. One instance per tenant per replica."""

    rate: float          # tokens added per second (this replica's share)
    capacity: float      # burst allowance
    tokens: float = field(default=0.0)
    updated_at: float = field(default_factory=time.monotonic)

    def allow(self, cost: float = 1.0) -> bool:
        now = time.monotonic()
        elapsed = now - self.updated_at
        self.updated_at = now
        self.tokens = min(self.capacity, self.tokens + elapsed * self.rate)
        if self.tokens >= cost:
            self.tokens -= cost
            return True
        return False


def replica_share(global_rate: float, healthy_replicas: int) -> float:
    """Divide the global budget. Recompute on membership change, not per request."""
    return global_rate / max(healthy_replicas, 1)

The important property is that the check touches no network and no lock beyond the per-tenant entry. It costs a few hundred nanoseconds and cannot be made unavailable by a dependency.

Push transformation to the edges

Every transform in the gateway runs on shared CPU. Ask whether it belongs to the client or the service instead. Response shaping for a mobile app belongs in a backend-for-frontend owned by the mobile team. Protocol conversion belongs in a sidecar. The gateway should own only genuinely cross-cutting transforms.

Use cells to bound the blast radius

Instead of one fleet fronting everything, run independent cells, each with its own replicas, config, and limit store, and route tenants to cells by a stable hash. A bad rollout that reaches one cell affects that cell's tenants only.

This costs real money and operational complexity, so it is an api gateway pattern for scale, not a default. It is also the only mitigation here that turns a total outage into a partial one, because it is the only one that breaks the correlation. The AWS Well-Architected reliability pillar covers cell-based architecture and blast radius reduction in depth.

Make health checks tell the truth

A health check that returns 200 whenever the process is alive is worse than none, because it gives the load balancer false confidence. Report unhealthy when worker pool utilization crosses a threshold or inbound queue depth exceeds a bound. Then make sure the load balancer will not drain every replica at once when they all report unhealthy together.

What to say in a system design interview

When asked to add a gateway, the weak answer is that it handles auth, rate limiting, and api gateway routing. That is a definition, and definitions do not demonstrate judgment.

The strong answer names the tradeoff: consolidating cross-cutting concerns removes duplication across services, and the cost is a correlated failure domain for all of them. Then get specific about one mitigation, ideally with numbers. Interviewers are listening for whether you understand that centralization is a trade, not a free win. The same instinct shows up in choosing between an API gateway and a load balancer and in deciding when microservices are worth their coordination cost.

Where to go next

Keep building the ingress picture with our comparison of gateways, load balancers, and reverse proxies, then edge caching and anycast routing for what sits in front, and caching strategies and their failure modes for what sits behind. More breakdowns are on the Levelop system design blog.