How Does a URL Shortener Work?

Key Takeaways

Ever wonder what happens when you click a tiny URL? You know those short links like Bitly and Tiny URL that somehow know exactly where to take you? Today, we're going to reverse engineer how URL shorteners actually work. And trust me, there's way more going on behind the scenes than you might think. So, what exactly does a URL shortener do? Simple. It takes a massive URL like this Amazon link that goes on forever and turns it into something short and clean. Click the short version and boom, you end up at the same place. But here's where it gets interesting. Let's say you're building the next Bitly. How many URLs do you think we'll need to handle? Try 100 million new URLs every day. That's over a thousand new short links created every single second. And since people click links way more than they create them, you're looking at over 10,000 clicks per second. Now think about storage. Over 10 years, that's 365 billion URLs you need to keep track of. Just storing the URLs themselves would take 36 terabytes. So the real question becomes, how do you even generate these short URLs? This is where the math gets fun. Short URLs can use numbers and letters. There's 62 possible characters 0 through 9, lowerase A through Z, and uppercase A through Z. But how short can we make them? Let's work backwards. We need 365 billion unique combinations. With one character, you get 62 possibilities. With two characters, 62 squared. That's about 3,000. With three characters, 62 cub about 238,000. Keep going, you get to seven characters. 62 to the 7th power. That's 3.5 trillion possible combinations. Way more than we need. So, seven characters it is. But how do we actually create them? There are two approaches and they couldn't be more different. The first approach just hash the long URL. Take any hash function like MD5, run the long URL through it and you get back a long string of random characters. The problem that string is way too long. Even the shortest hash give you 32 characters when you only want seven. So you take the first seven characters and call it a day. But wait, what happened when two different URLs give you the same first seven characters? You've got a collision. Now you're stuck. You have to try again with some variation of the original URL until you find seven characters that nobody else is using. Every time you want to create a short URL, you have to check if those seven characters are already taken. That's a lot of database lookups. The second approach is way more elegant. Instead of hashing, you just count. Every time someone wants to shorten a URL, give it the next number in sequence. URL number one, number two, number three, and so on. Then converted number to what's called base 62. Here's how that works. Let's say you're on URL number 11,157. To convert to base 62, you divide by 62 over and over, keeping track of the remainders. 11157 / 62 is 179, remainder 59. 179 / 62 is 2, remainder 55. 2 / 62 is zero, remainder 2. Now read the remainders backwards. 2 55 59. In base 62, two stays as two. 55 becomes t 59 becomes x. So URL number 11157 becomes 2tx. Your short URL is tinyurl.com/2tx. No collisions, no database lookups to check if it's taken, just clean math. To trade off, you need a way to generate unique numbers across multiple servers. That's its own engineering challenge and there's a security issue. If someone figures out your pattern, they can guess to next short URL. But for most cases, this approach is much cleaner. Now, generating the short URL is just half of the problem. The other half is what happens when someone clicks it. When you click a short URL, the system needs to look up the original URL and redirect you there. And this happens a lot more often than creating new short URLs. So speed matters. First check the cache. If the mapping is there, redirect immediately. If not, hit the database. Cache the result for next time. Then redirect. The redirect itself uses what's called a 301 status code. That tells your browser this URL has permanently moved to the other location. Your browser remembers this, so it might skip the URL shortener entirely next time. But here's what makes this really interesting. the scale. A single database can handle 10,000 lookups per second. So you need multiple database replicas to spread the load. Eventually you will need to split the data across multiple databases entirely. This is called shorting and is another whole engineering problem. You need to figure out how to distribute the data evenly, how to route requests to the right database, what happens when one database goes down, how to rebalance when you add more servers. And that's just the beginning. In the real world, you also need to think about rate limiting so people can't spam your service. You need analytics to track how many people click each link. You need security to block malicious URLs. What started as make this URL shorter becomes a lesson in distributed systems, caching, database scaling, and performance optimization. Every major tech company has built some version of this. Twitter shortens URL in tweets. Slack does it in messages. Even your company's internal tools probably do this. And the techniques we talk about, unique ID generation, caching strategies, database sharding, these patterns show up everywhere. Instagram uses similar ID generation for photos. Netflix uses similar caching for video meta data. Uber uses similar database splitting for trip data. The next time you click a shorten link, you will know there's a whole system working in milliseconds to get you where you're going. And you will start recognizing these same patterns in every app you use. Ready to ace your next technical interview? Join our community where we offer comprehensive courses on system design, coding, behavioral questions, machine learning, and object-oriented design. Learn more at bybico.com.