Shooting CS50 for MBAs, Cloud Computing, behind the scenes...

Key Takeaways

5 4 3 2 1 5 4 three two one begin the comments as to why the video is looping all right hello hello world this is cs50 we're here in Sanders Theater me and Lauren Scully getting ready for cs50 for NBAs uh cloud computing is the focus for today and it's when we wrap up this six lecture focus on computer science from the top down 104 so we're just getting ready here in sander s making a few adjustments to the camera uh we're today is going to be a very low-level discussion from the ground up as we focus on uh what the cloud actually is and what do we have here Scully um I need to write cloud computing on this so that it's very clear and post what we're editing and post would be what um when we edit the lecture okay so post production yes yes so Dan's gonna be doing the live cutting for today and then we will be doing a fine tuning okay and now we are we're looking here at camera one three two one three four one two three four three four nice and Al one does not have the light so our thoro now has a light on camera 4 which helps me know where to look he's camera one he's also camera one but camera four is right behind him so it's a common mistake that I make this is Hur I not going to be able to stand up if we if we keep doing this okay let me grab let me grab the yes sir do you remember how have we begun most of these right no no I've been standing here and then what do I say no oh I thought I turned it on you want it off or on oh all right quitting flux all right so we're putting some finishing touches we're going to be doing some drawing today so thanks to our friend bjor redig uh we have this uh drawing application that runs on a Surface connected to this big surface Hub screen we'll be using that in just a moment and now what is the role that this plays this makes sure we're in sync and it also gives us information and as to what episode we are filming I see and so by hearing the same sound on all four of our cameras that's what allows us to SN so it's a combination of the fact that when these come together and then we hear the sound that we know we're in syn very nice all right just about s we'll be ready in just one moment folks okay and we started both okay excellent starting on Ian or Melinda okay thanks it's the two sides coming together just about ready to begin oh David the only one who's not ready all right all right ready when you are all right so cloud computing it's this term that rather swept onto the scene in recent years and it sounds like it's some new and trendy technology but in reality it's really just a very nice packaging up of a whole number of technologies that have actually been with us for some time in fact cloud computing in its simplest form can really be thought of as just Outsourcing the hosting of your applications and really Outsourcing the hosting of your physical servers to someone else put another way renting space and renting time on someone else's computers but these days we just have so much computational capabilities that is our computers are so fast our CPUs are so many and we have so much RAM that new and fancier technologies have lent themselves to this trend of Hosting all the more software and putting all of the more Hardware offsite in the so-called Cloud so that companies both big and small can no longer uh no longer need to host their own physical Hardware or even a whole number of roles in their own local companies and so what we'll do now is dive into cloud computing look at some of the problems it solves look at some of the opportunities it affords but ultimately take a look from the ground up at what's underneath the hood here so that by the end of this we have a better understanding of what the cloud is why is it as useful and what it actually is not so with that said let's start with a simple scenario of course the the cloud perhaps uh deres its Origins from how the internet for some time was drawn which was just this big nebulous cloud in that it doesn't really matter what's inside that cloud although at this point you uh most surely uh appreciate that inside of this Cloud are are things like routers and uh running through those routers are packets both TCP IP and the like and underneath the hood then of this cloud is some transport mechanism that gets data from point A to point B so what might those Point A's and point uh b's be well if this here is my little old laptop connected somehow to the internet here and maybe down here there is some web server on which lives a whole bunch of web pages maybe it's my email maybe it's the day's news maybe it's some social media site or the like I at Point a want to somehow connect to point B down here now it turns out it's not all that hard to get a website up and running on the internet you can of course use any number of languages you can use any number of databases and you can do it with relatively little in uh experience just getting something on the Internet in fact it's not all that hard relatively speaking to get a prototype of your application or even your first version of your business up and running but things start to get hard quickly especially if you have have some success indeed a good problem to have is that you have so many customers and so many users uh hitting your websit that you can't actually handle all of the load now it's a good problem in the sense that this business is booming but it's of course an actual problem in the sense that your customers aren't going to be able to visit your website and buy whatever it is you're selling or read whatever it is you're posting if your servers can't actually handle the load and by load I simply mean the number of users per minute or per unit of time that your website is actually experienc Ing and its capacity by meanwhile would be the number of users it can actually support now why are there these limits in the first place well you may recall that inside of a computer is a CPU the brains of that computer and inside of a computer is some memory like RAM and there might be some longer term storage like hard disk space at the end of the day all of those resources and more are finite you can only fit so much physical Hardware in a computer uh humans have only in been able to pack so many resources into the physical space of a computer and then of course there's cost you might be able to only afford so much Computing capacity so if a computer can only do some number of things per second there's surely an upper bound on how many people can visit your website how many people can add things to their shopping cart how many people can check out with their credit card because you only have at the end of the day finite numbers of resources now what does that mean in real terms well maybe your web server can handle a 100 users per minute maybe it can handle a thousand users per minute maybe you can handle a thousand users per second or even much more than that it really depends on the specifications of your Hardware how much RAM how much CPU and so forth that you actually have and it also depends to some extent on how well written your code is and how fast or how slow your code your software actually runs so these are knobs that can ultimately be turned and through testing can you figure this out in advance by simulating traffic in order to estimate exactly how many users you might be able to handle at a time now the relevance to today is that the cloud so to speak allows us to start to solve some of these problems and also allows us to start abstracting away the solutions to some of these problems well let's see what this actually means so at some point or other especially when it's not just my laptop but it's like a thousand laptops or 10,000 laptops and desktops and phones and more that are somehow trying to access my server here at some point we hit that upper limit whereby no more users can fit onto my website per unit of time so what is the symptom that my users experience at that point if I'm over capacity well they might see an error message of some sort they might just experience a spinning icon because the website is super slow to respond and maybe it does respond but maybe it's 10 seconds later so at the end of the day they either have a bad experience or no experience whatsoever because my server can only handle so many requests at a time so what do you do to solve this problem if one server is not enough maybe the most intuitive solution is well if one server is sort of not giving me enough Headroom why don't I just have two servers so let's go ahead and do that instead of having just one server let's go ahead and have two and let me propose that on this second server is the exact same software so whatever code I've written and in whatever language it's written I just have copies of my website on both the original server and the second server okay now I've solved the problem in the simple sense that I've doubled my capacity if one server can handle a thousand people per second well then surely two servers can handle 2,000 people per second so I've doubled my capacity so that's good I've hopefully solved the problem but it's not quite as simple as that right at least pictorially I'm still pointing at just one of those servers so we're going to have to clean up this picture alone and somehow figure out how to get users or more generally traffic to both of these servers right I could just in naively draw an arrow like this but what does that actually mean we don't want to abstract away so much of the detail that we're ignoring this problem how do we implement this notion of choosing between left arrow and right arrow well let's consider what our Solutions might be if a user like me on my laptop is trying to visit this website and the website ideally is going to live at something like example.com or facebook.com or gmail.com or whatever I don't want to have to broadcast different names for my servers right and you might actually notice this on the internet you might notice if you start noticing the URLs of websites you're visiting especially for certain older stodger companies who haven't necessarily implemented this in the most modern way you might find yourself not just at www. something.com but if you look closely you might find yourself occasionally at www 1. something.com or wwww 2. something.com or even www13 doom.com which is to say that some companies appear to solve this problem by just giving different names similar names but different names to their two servers three servers 13 servers or however many they have and then they somehow redirect users from their main domain name www. something.com to any one of those two or three or 13 servers but this isn't very elegant the marketing folks would surely hate this because you're trying to build some brand recognition around your url why would you dirty it by just putting these arbitrary numbers in the URLs plus if you fast forward a bit in this story if for some reason down the road you get fancer bigger servers that can handle more users and therefore you don't need 13 of them you can get away with just six of them well what happens if some of your customers have bookmarked very reasonably one of those older names like www13 something.com so that now when they try to visit that URL gosh they might hit a dead end so you could solve that in some other way but the point is it would seem to create a problem quickly and it's just a naming mess why actually bother having your users see something as messy as these numbered servers it would be nice to do this a little more transparent so how could we do this well let me propose that we kind of need some middleman here so to speak whereby traffic comes from people like me on the internet and then either goes to the left or goes to the right or no matter how many servers we have goes to one of those actual web servers so how does this middleman and to borrow some past terminology how does this black box potentially work well let's consider some of the building blocks some of the the puzzle pieces we have technologically at our disposal now you may recall that every server on the internet has an IP address Internet Protocol address a unique address for it and that's again a bit of a white lie because there are Technologies via which you can have private IP addresses that the outside world doesn't see but let's uh stipulate for today's purposes that every computer on the internet certainly has an IP address whether public or private so maybe just maybe we could leverage an existing technology DNS the domain name system so that rather than only return one IP address of a server when you look up www.s something.com we return the IP address of the server on the left some of the time or the IP address of the server on the right some of the time effectively balancing our load our traffic across the two servers and in fact if you do this 50/50 you can take a really what's called a round robin approach and ideally uniformly distribute your traffic across multiple servers and what's nice in this model is that because you're using DNS the user doesn't really notice what's going on at the end of the day none of us humans really care what IP address we're actually going to if we visit facebook.com or gmail.com or the like we just care that our computer can find that server or servers on the internet so via DNS we could very cleverly via this middleman here which is really just going to be some third device some separate server it as a DNS device could just respond to requests from customers with either this IP add address or this IP address or any number of different IP addresses so does this solve the problem again most everything in computer science would seem to be off a trade-off at the end of the day and this seems almost too good to be true perhaps it's so simple it leverages an existing technology it just works so what prices might we pay well DNS it turns out gets cached quite a bit and what does cashing mean cashing something means keeping some past answer or more generally piece of information around so that you can access it more quickly the second and the third time and Beyond and so computers today Macs and PCs as well as servers on the internet other DNS servers on the internet for performance reasons will often remember the responses that they get from DNS servers for instance if on my Mac I visit facebook.com hypothetically a lot of times during the day it's kind of stupid if my laptop again and again and again and again ask some DNS server for facebook.com IP address if it already asked that same question an hour ago or more realistically two minutes ago or something like that it would be smarter if my operating system or even my browser Chrome or or Firefox or whatever I'm using actually remembers that answer for me so that my computer can just pull up that website Faster by skipping a step by not wasting time asking a server again for the IP address of a server and after all IP addresses it turns out just generally don't change that often it's certainly possible for a company or University or even a home user to change their computer's IP addresses but the reality is it doesn't change all that often the common case is to have the same IP address now as you might an hour from now or even a day or a week or a month from now but the key thing is that it can change and especially if you're worried about customers not just some personal website but you might lose business you might lose orders if users can't visit your website anything that puts your server up time so to speak being accessible on the internet at risk probably is worthy of some consideration so let me propose then that just one of these servers goes offline somehow maybe it's deliberate you need to do some service for it or maybe it crashed in some way or it got unplugged somehow or something went wrong such that now one or more of your servers across which you've been load balancing no longer can talk to the internet what might happen well if some customer's Mac like my own has remembered or cashed that particular server's IP address that is not a good situation because your Mac or PC or whatever is going to now try to revisit your website again and again and again at that old cached IP address that apparently can be a dead end and so even though you still have servers that could potentially handle that customer's request that customer's order that customer's desire to check out he or she really is still not going to be able to visit the website unless that cash expires maybe they reboot their computer so that the cash forcibly expires maybe they just wait some amount of time so that that IP address is forgotten by the browser or by the operating system or by some other DNS server until the new ones available IP addresses are picked up instead but there is that risk and I would argue that this risk is even higher especially for companies that might be considering moving their infrastructure from one service to another if you're deliberately going to move your servers from one IP address to another as might happen if if you change Cloud providers so to speak more on those in a minute really if you change the companies that you're using to host your servers your IP addresses will change and you certainly don't want to incur a huge amount of downtime in a situation like that so there are these trade-offs easy solution technologically pretty inexpensive to do it just works using existing technology but you open up yourselves to this risk all right so let's address that putting back the old proverbial engineering had it let's try to solve this problem it seems that giving a unique IP address to this server and to this server and any number of other servers that are back there might not be the smartest idea in so far as those IPS can get cached so what if we use DNS as follows when my laptop or anyone else's requests the IP address for dub. something.com why don't we return the IP address of this device here this load balancer as we'll start calling it where a load balancer is usually just a physical device or multiple physical devices who purp whose purpose in life is to balance load packets come in and similar in spirit to a router they do route information to the left to the right or some other direction but their overarching purpose isn't just to get data from point A to point B but to somehow intelligently balance that traffic over multiple possible destinations for point B identical servers in the case of our story here so what if instead we address this problem of potential downtime by returning the IP address of the load balancer and then by nature of private IP addresses or some other mechanism that the end user does not need to know or care about this load balancer somehow routes the traffic to either the first device or the second device lb here being our load balancer all right so we've seem to have solved this problem in so far as now we have configured our DNS servers to return the IP address of the load balancer there's no problem of downtime as we described a moment ago because if server won't goes offline for whatever reason no big deal the load balancer should hopefully just notice that and subsequently start proactively routing all incoming data that reaches its IP address to server two and not server one now how does the load balancer know well either a human could intervene maybe someone gets a late night call or text or page saying uh oh server one is down you better do something and then he or she can manually configure the load balancer to no longer send any traffic to server one that seems kind of stupid in an age of Automation and smart software Maybe we can do better and indeed we can a technique that's often used by servers is something uh modeled from the human world to use what you might describe as heartbeats to actually configure the load balancer and servers one and two to operate as follows maybe every second every half a second maybe every five seconds you configure server one and server two to send some kind of heartbeat message to the load balancer this is just like a tcpip packet some kind of network packet that's the equivalent of saying I'm alive I'm alive or more goofy like Bo boom bo boom there go the the heartbeat metaphor but the point is that one and two and any a number of other servers should be configured to just constantly reassure the load balancer that they are alive they are accessible they are ready to receive traffic and the load balancer similarly and you might see where this is going can very simply be configured to listen for that heartbeat and if it ever doesn't hear a heartbeat from server one or server two it should just assume that something's wrong the server has died it's gone offline something bad has happened so the load balancer subsequently should simply not route any traffic to that particular server until some human or some automated process brings the server back alive so to speak and the heartbeat resumes now of course this problem doesn't go away permanently if servers one and two stop emitting a heartbeat we really have no capacity for users but that would be an extreme scenario hopefully it's just one or few of our servers go offline in that way so we can configure our servers with these heartbeats which is think about it a very simple sort of physiologically inspired solution to a problem and even if it's not obvious how you implement it in code it really is just an algorithm a simple set of instructions with which we can solve this problem and yet damn it we've introduced a new problem and so this really is the old sort of uh leaky hose where just as we've plugged one leak or solved one problem another one has sprung up somewhere else along the line so what's the problem now what's the problem now the whole motivation of introducing server number two in addition to server number one was to make sure that we have enough capacity and better yet to make sure that if server one or server two goes offline the other one can hopefully pick up the load unless it's a super busy time with lots and lots of users visiting all at once so in fact the general idea at play here is high availability uh availability assum uh ensuring that if one server goes down you have other servers that can pick up the load being highly available means you can be tolerant to issues like that and then load balancing of course is just the mere process of splitting the load across those two end points but we have introduced another problem uh this might be abbreviated spof or more uh explicitly single point of failure and just as I've solved one problem by introducing this load B BC ER so have I introduced a new problem which is this there is now as you might infer from the name alone a single point of failure it's fine that I can now tolerate server one or server two going down but what can I not tolerate clearly what if the load balancer goes down so this is a very real concern maybe the load balancer itself gets overloaded maybe the load balancer itself has some kind of issue and if the load balancer goes down it doesn't matter how many web servers I have down here or how much money I've spent down here to ensure my high availability my server is offline if the single point of failure indeed fails now you'd like to think that the load balancer especially since it only has one job in life can at least handle more traffic than an individual server indeed clearly it must be the case that the load balancer is fast enough and capable enough to handle twice as much traffic as any individual server but that's generally accepted as feasible in so far as your website your real intellectual property is probably doing a lot of work talking to a database writing out files downloading things or any number of other features that just take more effort than just routing data from one server to another as a load balancer does but it doesn't matter how performant it is if the load balancer breaks goes offline for some reason your entire infrastructure is inaccessible so how do we solve this how do we go about an architect a solution to this well how did we address this issue earlier we addressed the issue of insufficient cap capacity or potential downtime by just throwing Hardware at the problem and so maybe we could do that same thing here maybe we could just introduce a second load balancer I'll call this lb as well and now we somehow have to I feel like we're just kind of uh endlessly going to be uh adding more and more rectangles to the picture but somehow we need to be able to load balance across now two servers and two load balancers so how do we do this well let me clean this up so that we have a bit more room to play with here and consider how a pair of load balancers might actually work so if my first server is here and my second server is here and I'm proposing now to have two load balancers one here and one here surely both of these have to be able to talk to both servers so we already have this necessity and somehow traffic has to come from the internet into this set of load balancers but probably only to one because we don't want to solve this with DNS and just have two IP addresses out there because if one breaks so we can recreate the same problem as before if we're not careful so what if we do this what if we use this building block of heartbeats in another way as well what if we ensure that our load balancers plural have just one IP address which at a moment ago seemed to create a single point of failure but what if we do this what if we also allow the load balancers to talk to to communicate over a network with each other so that one of the load balancers is constantly saying to the other I'm alive I'm alive I'm alive and so what the load balancers could be configured to do is that only one of them operates at any given point in time but if the other server the other load balancer no longer hears from that primary load balancer because of the heartbeats that are ideally both being emitted in both directions so that they can both be assured of the others up time if the secondary load balancer stops hearing the primary load balancer the secondary load balancer can just presumptuously reconfigure itself to take on that one and only IP address effectively assuming that the first load balancer is not going to be responding to any traffic anyway and the second load balancer can simply take on the entire load itself but the key difference now in this particular solution is that there's only one IP address that describes this whole architecture only one IP address between the two load balancers so we don't risk those potential dead ends that we had a little bit ago with our backend servers and so now it's starting to get more robust more highly available so that's pretty good we've solved most of these problems um we've sort of uh generously though uh swept one problem underneath the rug where by every time I draw another rectangle uh not just the first time but now the second time and add some interconnectivity somehow among them someone somewhere is spending some money and indeed I am solving these problems thus far by throwing money at the problem and frankly introducing complexity me and already look at how many arrows or edges there are now which might simply refer to physical wires which is fine but there's also a logical configuration that's now necessary and God forbid we have a third load balancer for extra high availability or any number of servers here 13 or 20 or 100 or a thousand servers it's a lot of cross connections not just physically but logically in terms of the requisite configuration so this complexity does add up and the cost certainly adds up and now once upon a time and not all that long ago if a company wanted to Arch architect this kind of solution you would literally buy two load balancers and you would buy two or more web servers and you would buy the requisite physical ethernet cables to interconnect the two and you'd probably buy a whole bunch of other Hardware that we've not even talked about like firewalls and switches and more but you would physically buy all of this Hardware you would physically connect all of this hardware and uh configure it to implement these several kinds of features but the catch is that the more and more Hardware you buy just probabilistically the more and more you invite some kind of failure maybe it's some stupid human error but more realistically one of your hard drives is going to fail and a hard drive has are typically rated for the Enterprise in terms of meantime between failure mtbf which generally means how much time how long should you expect a hard drive to work on average before it fails it breaks it just stops working so if you have a whole bunch of servers Each of which has a whole bunch of hard drives at some point combinatorically one or more of those drives is just going to fail which is to say you're going to have a problem and you're going to have to fix it yourself at some point too you're going to run out of physical space in fact perhaps one of the most constraining resources uh especially for startups is the physical space itself you probably don't want to start housing your servers in your physical office because you need a special room for it typically with uh enough cooling with enough access with enough electricity uh and any and enough humans to actually maintain it or you graduate from your own office space and go to a a data center a collocation facility whereby you maybe rent space in a physical cage with a a locking door inside of which you put racks of servers just uh racked up on big metal poles uh and you pack as many servers in there as you can but at some points you're going to B bumping up against other constrained resources physical space actual power uh capacity uh cooling as well as the humans to actually run this and so very quickly does operations Ops so to speak become an increasing cost and an increasing Challenge and one of the most alluring features of the Cloud so to speak is that you can move all of these details off site and you can abstract all of many of these let's say implementation details away whereby you yourself don't have to worry about the physical wires you don't have to worry about the make and model of servers that you're buying you don't have to worry about uh things actually breaking because someone else will deal with that for you but you have to still understand the topology and the architecture and the features that you want to that you want to implement so that you can actually configure them in the cloud so what do you actually get from cloud providers there's any number of them out there these days but perhaps three of the biggest are Amazon Google and Microsoft all of whom offer these days a very similar pallet of options and it's outright overwhelming if you visit each of their websites just how many Cloud products they offer but they would generally offer a number of standard products in the cloud for instance a virtualized server so you don't have to physically buy a server these days and plug it into your own ethernet uh connection your own internet connection in your own office you can instead essentially rent a server in the cloud which is to say that Amazon Google Microsoft or any number of other companies will host that server physically for you and they will take care of the issues of power and Cooling and if a hard drive fails they will go remove the old one and plug in the new one and ideally they will provide you with backup services but more sophisticated than that they can also help us recreate in software this kind of topology in other words even without having a human physically wire together this kind of graph so to speak that we've been building up here logically thanks to software these days you can implements this whole Paradigm not with physical cables not with physical devices but with software virtually what does that mean it means that humans over the past several years have been writing software that mimics the behavior of phys physical servers humans have been writing software that mimics the behavior of a router humans have been writing software that mimics the behavior of a load balancer and by implementing uh mimics the behavior of really we are just building in software what historically might have been implemented entirely in hardware and even that's a bit of an oversimplification because even when something is uh bought as Hardware there is of course software running on that Hardware that actually makes it do something but they're no longer dedicated devices you can use generic commodity PC server Hardware really and transform that Hardware into a certain role a backend web server a backend database a load balancer a router a switch and a number of other things and so what you are getting from companies like Amazon and Google and Microsoft and more is the ability to build up your infrastructure in software in fact uh the buzzword here the acronym is I AAS infrastructure as a service so you sign up for an account on any of those companies uh cloud services websites and you put in your credit card information or your invoicing information and you literally via command line tool so a keyboard or via a nice web-based graphical user interface guey do you point and click and say give me two servers and one load balancer or if you have enough money in the bank you say give me two servers and two load balancers configured for high availability or better yet you don't say any of that you just tell the provider give me a web Serv server and give me a load balancer and you deal with the process of scaling those things as needed in fact a buzzword dour is autoscaling which refers to a feature implemented in software whereby if a cloud provider notices that your servers are getting a lot of traffic business is good or it's the holiday season and you know you are bumping up against just how many users your one or two or three or more servers can handle autoscaling is a feature that uh will enable the cloud provider to just turn turn on virtually more servers for you so that you go from 2 to three automatically you can be uh happily asleep in the middle of the night and even though your traffic is peing doesn't matter your architecture is going to Auto scale and better yet especially financially if the cloud provider notices maybe 12 hours later oh all of your customers have gone to sleep we don't really need all of this excess capacity or maybe the holidays are now in the past you really don't need this excess capacity autoscaling also dictates that those servers can be virtually turned off so you're no longer using them you're no longer load balancing to them and most importantly you're no longer paying for them so this is a really really nice value ad at this point there's no human crawling around on the floor rewiring things and plugging in new servers there's no Finance person having to approve the PO to actually order more servers just to increase your capacity and most importantly there's no latency between the time when you notice oh my God we're getting really successful and can't handle our load uhoh it's going to be a two three we lead time before we can even get into more servers thanks to cloud computing you can literally log in Amazon's Google's Microsoft's website and click click click have more server capacity within seconds within minutes far faster than the physical world traditionally allowed so those are just some of the features now that we gain from Outsourcing to the so-called Cloud so where does some of this capability come from well it turns out that over the past many years uh humans have been getting better and better and better at packing more physical Hardware into the same form factor into the same physical space So at the level of CPUs the brains of a computer we humans have gotten much better at packing more and more transistors for instance onto a CPU and transistors are little switches that can turn things on and off zero and one one and zero so you can store more information and you can do more with that information more quickly uh CPUs today also have more cores which you can think of as many CPUs inside of the main CPU so that a computer with multiple cores can literally do multiple things at a time but the funny thing is that we humans over the past you know decade or two really haven't been getting fundamentally faster at life uh at the end of the day I can only check my email so quickly I can only post on Facebook so quickly I can only check out from Amazon uh so quickly because we humans have of course a sort of uh finite speed to ourselves we're not just getting we're not doubling in speed all Moore's Law every year or two so we have it would seem a lot of excess Computing capacity these days computers are getting so darn fast we don't necessarily know what to do with all of these CPU cycles and with all of the ram that we can fit into the same physical box at half of the price that it cost us last year and so manufacturers and companies realized that hm we could actually build a business on this increased capacity we can you know implement the human the uh the computer equivalent of time sharing so to speak which has long been with us in the history of computing but we can do this on a much more massive scale Now by taking one physical server that has maybe two CPUs or 16 CPUs or 64 CPUs and maybe gigabytes uh gigabytes and tens of gigabytes or hundreds of gigabytes of RAM all inside of the same physical device plug it into an internet connection and then run special software on that one server that creates the illusion that there's multiple servers living inside of that box and this virtualization software uh is implemented by way of software called a virtual machine or virtual machine monitor or another word might be hypervisor there's different ways to describe essentially the same thing but a virtual machine is a piece of software running on a computer inside of which is running some other operating system typically so you you might have one server running Windows but inside of that server are multiple version uh multiple virtual machines Each of which itself is running Windows so you might be able to chop up one computer into 10 or even into a 100 or perhaps more commonly you might have a server running Linux or some Unix based operating system also with virtual machines on it but those virtual machines might be running Linux themselves or Unix or Windows or any number of versions of Windows and so this is the beauty when you have so much excess capacity and so much uh so many available CPU cycles and so much RAM you can kind of slice that up and then sell portions of the server's capacity to customers and if you're really clever you might look at your customer usage patterns and realize that you know what uh it's not necessarily as simple as just taking my server and dividing it up into n different slices where N is a generic variable for number and then selling it or renting that space really to end customers because you know what some of those customers might have some booming businesses which is great but some of those customers might not have many users maybe it's a a few dozen maybe it's a few hundred but it's really a drop in the bucket so instead of selling my Compu uh my Computing resources to just end customers maybe I'll sell it to twice as many customers or three times as many customers and essentially oversell my server's capacity but H expect that on average this is just going to work out because some customers will be using a lot of those Cycles because business is good and some won't be because it's just they don't have many customers or merely it's a personal website that doesn't get much usage anyway and so for some time um has there of course been this risk when you sign up for a web hosting company or a cloud provider that your website actually might get really slow for reasons outside of your control if you are Co co-located on a server that some other booming businesses on your users might actually suffer if your web host has oversold it so and so in fact you sort of this is one of those situations where you get what you pay for if you're Googling around and finding uh various Cloud providers or web hosting companies more specifically you might be able to find a deal like $10 per month or $50 per month as opposed to a 100 or 200 or more dollars per month and you do get what you pay for because those Fly by Night operations that are selling you uh space and capacity super cheaply probably are overselling and overc committing so these are the trade-offs too just how much money do you want to save versus how much risk do you actually want to take on generally it's safer to go with some of the bigger fish these days certainly when building a business uh as you might on a company like Amazon or Google or Microsoft or derivatives thereof so just to paint a more Concrete technical picture of what virtualization is here's a picture as you might think of it so you have your physical infrastructure here so that's the actual server from Dell or IBM or whoever then you have the host operating system which might be windows but is often uh Linux or some variant of Unix instead and then you have the hypervisor this is the piece of software that you install on your server that allows you to run multiple virtual machines on top of it and those virtual machines can each run any number of different operating systems themselves or even different versions of operating systems and so depicted here up top are the disperate guest os's operating systems that might be on there maybe this is Linux and Solaris and this is Windows itself or any number of other combinations whatever your customers want or whatever you want to provide or uh essentially rent to customers you can install but you do pay a price so as beautiful as this situation is and as clever as it is that we're leveraging these excess Resources by slicing up one server into the illusion of in this case three or more generally more there's some overhead because this hypervisor has to be a middleman between your guest operating systems and your host operating system the one actually physically installed the server and any layers of IND Direction like this so to speak have got it cost you some amount of time if there's some work being done here and you only have a finite number of resources the hypervisor itself is surely consuming some of your resources and gosh this just seems really inefficient especially if all of your customers are using the same operating system my God why do you have to have copies of the same OS multiply installed this just doesn't feel like it's leveraging uh much economy of scale and so it turns out there's a a newer technology that's gaining Steam and this is known not as virtualization per se but containerization the most popular instance of which is perhaps a company called Docker and the world of Docker is a little shorter it's a little smarter about how resources are shared you still have your infrastructure your physical server and you still have your host operating system whether it's Linux or or uh Unix or something like that but then instead of a hypervisor you have the docker engine which is really just an equivalent of that base layer of software but notice what's different in this case here we've kind of collapsed the previous picture in fact thanks to our friends at Docker who put this together here the guest OS has disappeared and you instead have your different applications and your different binaries and libraries as this abbreviation means all running on the docker engine now what does this mean this means when running Docker you typically choose your operating system for instance uh Ubuntu Linux or Debian Linux or something else Al together and then you essentially share that one operating system across multiple containers instead of virtual machines we now have containers so in other words you ensure that your different slices all share some common software the kernel so to speak the the base core of the operating system but then you uniquely layer on top of that base system that base set of default files whatever customizations your customers or you yourself want but you share some of the resources and long story short what this means is that containers tend to be a little lighter weight there's less waste of resources because there's less overhead of running them which is to say that you can generally start them even more quickly and better yet you can still isolate your different products and your different Services database and web server and email server and any number of other features all within the illusion of their own installation their own operating system even though there are some shared resources here so this too has been made possible by the capabilities of Modern Hardware and the cleverness frankly of humans in actually finding Solutions or creative uses for those available resources but what other features or topics come into play in this world of of cloud computing we talked about availability and and caching and and costing really figuring out where we're going to actually spend our money by throwing Hardware at problems and scaling more generally but there's also issues of replication which actually do relate to high availability so to speak but replication refers to duplication of data and really backups more generally as a topic and then there's also some other funky acronyms that are very much invogue these days besides infrastructure as a service there's also platform as a service paas or software as a service sa now saas even if you've not used it under this name odds are you have been using it if you do use G Gmail or outlook.com or any web-based email service you are using software as a service you don't really know or need to care where in the world your emails physically live or how many servers they're spread across or how your data is backed up or for that matter when you click Send how the email even gets from point A to point B you are treating Gmail and Outlook as a software as a service with all of the underlying implementation details abstracted away you just don't know or care how it's implemented well at least least if uh everything's working you probably do care if uh something goes down but there's this intermediate step between this extreme form of abstraction where all you see is just the top level service and the lowest level implementation that we've discussed which is infrastructure as a service whereby when using something like Amazon you literally click the button that says give me a load balancer you literally click a button that says give me two servers you literally click a button that says give me a firewall or any number of other features so Amazon and Microsoft and Google to some extent have all implemented these low-level services that still require that you understand the technology and you understand networking and you understand scaling and availability but you so much more easily and inexpensively and efficiently literally with just a laptop or desktop without any data center of your own stitch together the topology or the architecture that you actually want albe it in the cloud platform as a service though has arisen as sort of a middle ground here whereby you might have services like Heroku which you might have heard of which themselves actually run on infrastructures like uh Amazon or Google or Microsoft or the like but they provide themselves a layer of abstraction that isn't quite as high level so to speak as what you get from software as a service in fact these platforms as a service don't provide you with applications they just make it easier for you to run your applications in the cloud now what does that mean well it's all fun and exciting to understand load balancing and understand networking and understand the need for multiple servers and the entire conversation that we've had thus far but at the end of the day if I'm a software developer or I'm trying to build a business all I care about is making my internet application available to real users I really don't care about how many servers I have how many databases I have how the load balancers talk to one another like that's all fine and intellectually interesting but I just want to get real work work done so I'm willing to pay a bit more for this I'm willing to pay some middleman like a Heroku or any number of other services a platform as a service to abstract away those kinds of details so I have the wherewithal and I have the willingness to actually say Host this as a web server so give me a web server I will pay you some number of dollars per month to give me a web server but I want you Heroku to deal with the autoscaling of it I don't care how many servers it is I don't care how they are connected I don't care anything about these heartbeats I just want to have the Illusion for my own sake of just one server that somehow grows or shrinks dynamically to handle my customer base meanwhile things like load balancing I just want the my customers to be able to reach my server I don't care how it's implemented I don't care how it's made to be highly available I just want that to work and so C uh companies like Heroku provide these platforms as a service that just make your life a little bit easier and you don't have to think about or know about or worry about as many these details now to be fair if something breaks you might not understand exactly what's going wrong and you yourself might not be able to solve it indeed you might be entirely at the mercy of the cloud provider