Database Sharding Explained | Range vs Hash vs Directory Sharding

Key Takeaways

Welcome to design. Today we're breaking [music] down one of the most important concepts in large scale system design, database chararda. At the simplest level, chararda is about survival. When a single monolithic database can no longer handle growing data size, read traffic, or write throughput, vertical scaling stops working. You can't keep adding CPU or memory forever. Sharding solves this by splitting data horizontally across multiple independent databases called shards. Each shard holds only a subset of the total data, but together they represent the complete system. Once data is distributed, reads and writes happen in parallel. Throughput scales almost linearly and failures are isolated instead of taking down the entire system. There are multiple ways to shard data and each comes with trade-offs. Rangebased sharding splits data by value ranges such as price, user ID ranges, or timestamps. This is simple and efficient for range queries, but has a major weakness. If traffic concentrates on a specific range, you end up with hot shards, uneven load, and degraded performance. Keybase sharding, also known as hashbased sharding, uses a hash function on the shard key to determine where data lives. This gives excellent load distribution and avoids hot spots. However, it breaks natural ordering, which makes range query slower and more complex. Directory-based sharding uses a lookup service that maps attributes like region or tenant to specific shards. This approach is highly flexible and allows dynamic rebalancing, but introduces an additional dependency. If the directory becomes slow or unavailable, the entire system is affected. In practice, large-scale systems often combine multiple sharding strategies to balance performance, flexibility, and operational complexity. The most critical decision in any shard system is choosing the shard key. A good shard key must have high cardality so data spreads evenly. It should have uniform access frequency to avoid hotspots and it should not be monotonically increasing. Keys like time stamps or auto increment IDs may seem convenient but they silently create unbalanced shards and bottlenecks over time. Once data is shard aid every request must be routed correctly. This can be done in three ways. Shardaware clients route requests directly but are tightly coupled to shard topology. A dedicated routing tier centralizes shard logic and offers clean abstraction at scale. Shardaware nodes sit in between providing a balance of flexibility and control. Most modern architectures prefer a routing tier for long-term evolvability. Finally, sharding is almost always combined with replication. Each shard has a leader that handles rights and multiple followers that serve reads. Leaders are distributed across nodes, so no single machine becomes a bottleneck. This design delivers high availability, fall tolerance, and horizontal scalability, but it also introduces coordination and consistency challenges. Sharding is not just a database optimization. It's a foundational architectural decision that impacts APIs, query design, consistency models, and operational complexity. Master it, and you're no longer just building applications. You're designing distributed systems.