System design interviews test how you reason about tradeoffs at scale, not memorized answers. These are the core concepts every candidate should be able to explain and apply to a live design problem.
Start by clarifying functional requirements (what the system must do) and non-functional requirements (scale, latency, consistency needs) before designing anything. Estimate rough scale (requests/sec, data volume) to inform architecture choices, sketch a high-level design (API, data flow, major components), then go deep on the 1-2 hardest parts the interviewer probes — data model, bottleneck component, or a specific tradeoff — while narrating your reasoning throughout.
Vertical scaling means adding more resources (CPU, RAM) to a single machine — simple but has a hard ceiling and creates a single point of failure. Horizontal scaling means adding more machines and distributing load across them, which scales further and improves fault tolerance, but requires the application to handle distributed state, load balancing, and coordination, which adds real complexity.
In a distributed system experiencing a network partition, you must choose between Consistency (every read sees the latest write) and Availability (every request gets a response, even if it might not be the latest data). You can't guarantee all three of Consistency, Availability, and Partition tolerance simultaneously during a partition — since partitions are a fact of distributed systems, the real-world choice is between CP systems (e.g., HBase, or a distributed relational database with synchronous replication, which may reject requests during a partition to preserve consistency) and AP systems (e.g., Cassandra or DynamoDB, which stay available and resolve conflicts after the partition heals).
A load balancer sits in front of multiple backend servers and distributes incoming requests across them to avoid overloading any single instance and to enable horizontal scaling. Common algorithms include round robin (cycle through servers evenly), least connections (send to the server with the fewest active connections), and IP hash (route based on client IP for session stickiness). It also typically does health checks to route around failed instances.
Caching helps when the same data is read frequently relative to how often it changes, reducing load on the primary data store and cutting latency. Common strategies: cache-aside (app checks cache first, falls back to DB and populates cache on miss), write-through (writes go to cache and DB together), and TTL-based expiration (data automatically expires after a set time) — the hardest part is invalidation, ensuring the cache doesn't serve stale data after an underlying write.
SQL/relational databases enforce a fixed schema, support strong consistency and complex multi-table joins/transactions, and suit data with clear relational structure. NoSQL databases (document, key-value, wide-column, graph) typically trade some consistency/join capability for flexible schemas and easier horizontal scaling, suiting high-throughput, loosely-structured, or rapidly-evolving data. The choice depends on your consistency requirements, query patterns, and expected scale — not one being universally 'better'.
A message queue (like Kafka, RabbitMQ, or SQS) decouples producers and consumers — a producer publishes a message without needing the consumer to be available or fast, and the consumer processes at its own pace. This absorbs traffic spikes (the queue buffers load), improves fault tolerance (a consumer crash doesn't lose messages), and allows async processing so the producer's request doesn't block on downstream work.
Identify the actual bottleneck first — it's rarely the same layer across systems. Common levers: add caching to reduce database load, introduce horizontal scaling with a load balancer for stateless services, shard or replicate the database for read/write scaling, move slow synchronous work to async queues, and add a CDN for static content. The key interview signal is reasoning about which bottleneck to address first based on where load actually concentrates, not applying every technique at once.
Sharding splits data across multiple database instances, typically by a shard key (hash of user ID, geographic region, or a range of values), so no single node holds the entire dataset. Hash-based sharding gives an even distribution but makes range queries and resharding painful, since adding a shard reshuffles most of the key space; range-based sharding keeps related data together and eases range scans but risks hot spots if traffic skews toward one range. The bigger challenges are cross-shard queries and joins (usually pushed to the application layer), maintaining transactional consistency across shards, and rebalancing data as the cluster grows without downtime. Because of this operational cost, most systems delay sharding until vertical scaling and read replicas are genuinely exhausted.
A read replica is a full copy of the primary database that serves read traffic, used to scale reads horizontally and provide failover, whereas a shard holds only a subset of the data and is used to scale both reads and writes. Replication lag is the delay between a write committing on the primary and that write becoming visible on a replica, caused by the time it takes to ship and apply the replication log asynchronously. This lag means a client can write data and then immediately read from a replica and not see its own write, which is a common source of confusing bugs. Systems mitigate this by reading from the primary right after a write (read-your-writes), routing a given user consistently to the same replica, or using synchronous replication for critical paths at the cost of higher write latency.
Consistent hashing maps both data keys and servers onto the same hash ring, so a key is owned by the first server found going clockwise from its hash position. Its advantage over naive hashing (like key % N servers) is that adding or removing a server only remaps the keys between that server and its neighbor, rather than reshuffling almost the entire keyspace. It's used in distributed caches (like client-side sharding for Memcached) and distributed databases or load balancers to minimize data movement and cache misses when the cluster size changes. In practice, systems also use virtual nodes — mapping each physical server to multiple points on the ring — to smooth out uneven load distribution.
Token bucket works by adding tokens to a bucket at a fixed rate up to some capacity, and each request consumes a token; if the bucket is empty the request is rejected or queued, but the bucket can absorb short bursts up to its capacity. Leaky bucket instead processes requests at a fixed, constant output rate regardless of how bursty the input is, smoothing traffic but not allowing bursts through. Fixed-window and sliding-window counters are simpler alternatives that count requests in a time window, but fixed windows allow traffic spikes at window boundaries, which sliding windows correct for at the cost of more bookkeeping. The right choice depends on whether you want to tolerate bursts (token bucket) or strictly smooth traffic (leaky bucket), and rate limiters are typically implemented against a shared store like Redis so limits apply consistently across multiple API server instances.
A CDN (content delivery network) caches content at edge servers physically closer to users, reducing latency and offloading traffic from the origin server, and it's most valuable for static or infrequently-changing content like images, video, JS/CSS bundles, and increasingly cacheable API responses. On a cache miss, the edge node fetches from the origin, serves the response, and caches it according to headers like Cache-Control and ETag, which control TTL and revalidation behavior. CDNs also absorb traffic spikes and provide a layer of DDoS mitigation, since attack traffic often gets absorbed before it reaches the origin. The tradeoff is staleness: invalidating or purging cached content across globally distributed edge nodes is slower and harder than invalidating a single cache, so CDNs are best suited to content that tolerates some delay before updates propagate.
An idempotent operation produces the same end result no matter how many times it's executed with the same input — calling it once or five times leaves the system in an identical state. This matters because distributed systems have unreliable networks: a client might time out waiting for a response and retry a request, and if that request wasn't idempotent (like 'charge $10'), the retry could double-charge the user. The common pattern is to have the client generate a unique idempotency key per logical operation, and the server stores the result of that key's first execution so a retry with the same key just returns the original result instead of re-executing it. GET, PUT, and DELETE are naturally idempotent by HTTP convention, while POST is not, which is why idempotency keys come up most often in payment and order-creation APIs.
An index speeds up reads that filter or sort on the indexed column by avoiding a full table scan, typically turning a linear lookup into something closer to logarithmic via a B-tree structure. The cost is on the write path: every insert, update, or delete has to also update every index on that table, so more indexes mean slower writes and more storage overhead. At scale this tradeoff becomes a real capacity planning decision — over-indexing a write-heavy table can bottleneck throughput, while under-indexing a read-heavy table causes slow queries and full scans that hammer the database. The practical approach is to index based on actual query patterns observed in production, monitor slow query logs, and periodically remove indexes that aren't being used.
A monolith is a single deployable unit where all functionality shares one codebase and often one database, which makes local development, transactions, and debugging simpler because everything runs in one process. Microservices split functionality into independently deployable services communicating over the network, letting teams scale and deploy components independently and use different tech stacks per service, but this introduces distributed-systems complexity: network calls can fail, cross-service data consistency needs patterns like sagas instead of simple transactions, and you need service discovery, monitoring, and versioning across many moving parts. The decision is less about which is 'better' and more about organizational scale — microservices pay off when independent teams need to ship independently and components have very different scaling profiles, whereas a small team usually moves faster with a monolith. Many systems also land in between, starting monolithic and splitting out specific services only once a clear scaling or ownership bottleneck justifies the cost.
An API gateway sits in front of a set of backend services and acts as the single entry point for clients, routing each incoming request to the appropriate downstream service based on the path or host. It centralizes cross-cutting concerns that would otherwise be duplicated in every service — authentication and authorization, rate limiting, request logging, and TLS termination — so individual services can stay focused on business logic. It's also commonly used to aggregate or transform responses from multiple services into a single response for the client, reducing round trips for consumers like mobile apps. The tradeoff is that the gateway becomes a critical piece of shared infrastructure, so it needs to be highly available and low-latency, since every request now flows through it.
Strong consistency guarantees that once a write completes, every subsequent read from any node sees that write immediately — simpler to reason about, but it usually requires synchronous coordination across replicas, which adds latency and can reduce availability during network issues. Eventual consistency guarantees that if no new writes occur, all replicas will eventually converge to the same value, but there's a window where different nodes can return stale data for the same key. In practice, the choice is per-use-case rather than system-wide: a bank balance update usually needs strong consistency, while a social media like-count or view-count can tolerate eventual consistency in exchange for lower latency and higher availability. Many real systems mix both, using strong consistency for critical writes and eventual consistency for less critical, high-volume data.
A circuit breaker wraps calls to a downstream service and tracks failures; once failures cross a threshold, it 'trips' open and starts failing fast (or falling back to a default) instead of continuing to send requests to a service that's clearly struggling. This protects the calling service from wasting resources on requests likely to time out or fail, and just as importantly protects the downstream service from being overwhelmed further while it's trying to recover. After a cooldown period the breaker moves to a half-open state and allows a small number of test requests through to check if the downstream service has recovered, closing fully if they succeed. It's most useful for synchronous service-to-service calls in a microservices architecture, where one slow or failing dependency can otherwise cause cascading failures and resource exhaustion, like thread pool starvation, across the whole call chain.
Service discovery is the mechanism by which a service finds the network location (IP and port) of another service it needs to call, which is necessary because instances in a distributed system are constantly being added, removed, or rescheduled, so hardcoding addresses doesn't work. In client-side discovery, the calling service queries a registry (like Consul, etcd, or ZooKeeper) directly and picks an instance itself; in server-side discovery, the client calls a load balancer or gateway that queries the registry on its behalf. Services typically register themselves on startup and send periodic heartbeats, so the registry can remove instances that stop responding and avoid routing traffic to dead nodes. In containerized or Kubernetes environments, much of this is handled implicitly through DNS-based service discovery tied to the orchestrator's own service registry.