Keynote: Scaling challenges in cloud networking

Key Takeaways

hi my name is dave malt welcome to my talk on scaling challenges in cloud networking i'm a technical fellow here at microsoft and the engineering lead for the azure networking team and the work we'll be talking about today is the work done by that team collectively so before i get going i'd like to share with you some notion of the scale that we're talking about why because cloud networking and cloud computing is all about building massive shared infrastructure and then virtualizing it splitting it up across many many tenants so that each one can have a customized experience to them and that's where a bunch of the scaling challenges are going to come from so let's start by giving you some sense of the physical size i wish i could take each of you out to a data center and show you what it's really like but because i can't do that here's an architectural rendering of one of our data center buildings that gray area in the middle is where the it equipment the servers and the network equipment would be located now we don't actually park 737 jets inside of our data centers but those two 737s are shown there to give you some sense of how large that physical space is now imagine walking around those two 737s in a space that is completely filled with computers and when we build our data centers again to give you a sense of scale we don't build just one of those we build lots of them this is an aerial view of a relatively small set of data centers that we've built in just one of microsoft's regions around the world each of those white boxes you see there is just like i showed you in that previous picture big enough to have two 737s parked in it now of course what are those spaces actually filled with well they're filled with row after row of computer equipment computer equipment network equipment the fiber that it takes to turn that into that massive shared infrastructure now another way to talk about the scale of our networks is by looking at our global network so microsoft operates 61 regions all around the world each one of those regions is going to have multiple data centers inside of it so we can have availability zones to offer our customers more resilience to their services now those 61 azure regions are connected by over 175 000 kilometers of fiber that includes both terrestrial fiber and subsea cables many of which were actually built custom for microsoft in addition to those 61 regions we have another 185 what we call network edge sites our goal is to get microsoft presence within 25 milliseconds of all the eyeballs on the planet we have an additional 200 of what we call express route partners basically providers of local loops that we work with in order to hook up customers and their enterprise facilities their stores their offices to this microsoft network and we do that at over 20 000 peering locations so when we talk about the scale of cloud computing that's what we mean now those may be impressive numbers but you might be asking yourself what does this have to do with research where are the actual challenges that come up from trying to build and operate a network that operates at the scale i'll walk you through a couple of those challenges now one is just the sheer rate at which growth happens the past 20 months for us as well as probably for everyone listening to this have been astounding and unusual as uh society shifted online in response to the coven 19 pandemic we at microsoft saw the traffic demand on our wide area network quadruple in the space of 20 months that meant we had to rapidly grow and expand our network in order to provide capacity for all those new services all those new demands that society was making as it shifted uh dramatically and quickly online we saw the number of routes handled by our network grow 30 percent in only the past six months and what makes that challenging is that every customer coming to a cloud hosting facility is unique is different has different demands and every customer wants an optimal solution for their particular service their particular workload that makes traffic engineering that makes planning for these kinds of networks extremely difficult give you a couple examples why some of our customers are the extreme of mice so for example we have customers out there that might have 30 megabits per second of traffic but it's divvied up over 300 000 different endpoints three hundred thousand different routes that means making building traffic engineering and policy systems that can manage those things extremely difficult just to do the sheer scale that they have because of course we have many many customers like this we have other customers who are the epitome of the elephant flow customers who might have 200 gigabits per second of traffic but divided up only over a small number of endpoints a small number of routes that creates the opposite challenge to traffic engineering and planning how can we keep these very large elephant flows from causing congestion in our network or interacting with other people's flows that are flowing on our network and the rate of growth that quadrupling in only 20 months itself creates all sorts of challenges the reality is that these networks have gotten too big for human beings to think about them they're just too large and there are so many different constraints on the network and that'll impact how we grow it out so for example the network has to be reliable that means keeping track of the mean time to between failures the mean time to repair for all the different fiber routes on the network so we can predict ahead of time what the availability of every site on the network is going to be we have to keep track of all the workload demands what they are today and what they're likely to become going forward into the future we need to keep track of the topology not only because we're always adding new regions but because we're always innovating we're thinking of new ways to enhance our topology new ways to scale out better we're merging new networks into ours so the change comes from both external as well as internal factors and ultimately we need a way that we can rapidly iterate why because even the metrics we're trying to optimize we're continually learning more about those metrics and learning about what features the network are important and we need to be able to change our optimization criteria to do that that create a very uh unique and rich set of demands for a new system that we had to build so together working with microsoft research we built a system we called orcas is basically a software defined network planner it's fully automated can basically solve the capacity that we need for the next 12 to 18 months by taking into account the demands both forecasted and historical the current utilizations on our network the current topology plan topology changes and shifts the various transformations that we expect these all get fed into a compiler we can run simulations of what we think uh the network will be that comes out of the topology compiler to see if that output network still is going to meet all of our safety and reliability requirements and then of course we have to compare that predicted future network that we're going to need against the network we already have and identify where we're going to have to augment extend build new routes physically out there in the world to make the network we have look like that network we predict that we're going to need and that then becomes build signals where we actually go out and physically acquire those assets this is always one of the aspects of networking that's always fascinated me that in networking we have to optimize with the constraints of the real world in mind we're sort of joining the logical world of software with the physical world of reality out there sometimes we we might want to get a fiber path that goes between two locations but it's just not available i'm going to change gears a little bit now and talk about another kind of challenges that we have in the network and that is the challenge of providing uh scalable performance many of you are probably already familiar with how tcp moves data between applications running on different servers connected to a network so for example when an application shown by that orange box on the left wants to send data to another application running on another server it's going to begin by taking that data and posting it down into the kernel there the tcp stack will take the data turn it into pages called segmenting it and then indicate each of those segments each of those packets to the network interface controller the nic which will then transmit them out to the network indicated by a single switch here in this figure the network will take those packets deliver them over to the nic on the receiving host where again as each set of packets arrived and interruptedly posted to the cpu which will have to escape into the kernel where the tcp stack will take the data from those packets combine them together and eventually producing a stream of data which is made available to the application but actually get it to the application it then has to be copied out of the kernel buffers and into the user space memory where the application can get at it the result of all this is a very heavy demand on cpu and concurrent with that a very high latency typically we see 10 to 20 percent cpu utilization in order to move 40 gigabits per second worth of data from one server to another and even when those servers are co-located in the same physical rack latencies are typically 10 to 20 microseconds just due to all the intro processing and cpu scheduling that's necessary to get the data moved there is another way of course one of those ways is remote direct memory access or rdma as indicated at the bottom of this figure what rdma leverages is specialized hardware inside the neck on the server what happens with rdma is when an application wants to send data it simply posts a signal to the neck saying i'd like you to copy the data from the following pages right out of my virtual memory to these pages in virtual memory of a receiving application go do it that's all that's needed from the sending application from then on that nick in the server will take care of dmailing the data out of the main memory of the sending server packetizing it transmitting it across the network the nik on the receiving server will similarly take care of receiving the data dmailing it back into the virtual memory of the receiving application and when all that data has been received it then posts a single indication to the receiving application saying hey the data's there you can go process it the impact of this on cpu and latency is pretty dramatic we see that with rdma transferring 40 gigabits per second takes less than one percent of the cpu and typically has about two microseconds of latency so what does this mean well for azure there are lots of potential savings just in terms of the amount of cores that we're no longer spending on cpus essentially doing just data motion for customers people hosting on azure benefits again cost savings they have fewer cores that they're spending but also lower latency we find that we can get to less than one millisecond read write latency when the transfers are happening over rdma so this is a really nice benefit but getting there is tough why because to really make use of rdma we need what we call hundred by hundred by hundred we wanna be able to deliver a hundred gigabits per second of rdma traffic between two servers located up to a hundred kilometers apart and at a hundred percent availability hundred percent reliability why because what we use rdma for inside of azure is storage applications communication between storage servers communications between storage servers and the compute servers that host virtual machines we're actually serving those blocks of virtual hard drives from the storage cluster over to the vms over rdma as a result we've seen a very interesting traffic shift over 70 percent of the traffic in our network is now rdma tcp quick all those protocols that people usually think about as dominating uh network traffic are actually a distance second in size inside of the azure network we have you know 10 to the 18 bytes per day being transferred by rdma as part of 10 to the ninth i o operations happening per day we've got probably the largest rdma deployment in the world and we're completely dependent upon it we've recognized so many cpu savings by moving things to rdma that if we ever tried to fall back to tcp or some other protocol for transferring the data we'd run out of cpu in order to handle those now getting to this again was not easy there are a bunch of actual research challenges that had to be solved as well as some interesting deployment ones i like to use the example or the metaphor of race cars and minivans tcp is sort of like a minivan it's pretty easy to drive it can carry small things it can carry lots of people doesn't require a lot of expertise to use rdma is kind of like an f1 race car it is a highly optimized engine that it gives you very high performance essentially what we're trying to do is take a road system that was designed that has traditionally been used for minivans and instead make almost all the cars on it f1 performance race cars that are trying to get to their destinations as fast as possible that created some really interesting challenges to solve rdma to work well requires very low loss on ethernet and ethernet is typically a lossy physical medium making it lossless requires using something called pfc priority flow control now there were theoretical potential that when using pfc there could be things like deadlocks or unfairness would arise in networks what we found as we built larger and larger rdma networks is that those weren't just theoretical we actually saw them happen this figure on the lower left shows a deadlock that can actually happen inside of a set of switches running in the network that we discovered in practice and was able to rectify and fix working again uh with a bunch of research collaborators we came up with new congestion control protocols like dc qcn that make rdma work well over ethernet networks when running over long distances like 100 kilometers running over long distances is actually quite tricky why because as you go over longer distances the speed of light delays actually increase and add up and that means that the feedback that a sending switch receives is very it takes longer and longer for feedback to get between two switches and switches have a limited amount of buffer space so figuring out how much of that buffer space to allocate to storing rdma traffic while it's in flight across the network is a really hard optimization problem we did a lot of detailed modeling in order to figure out how to solve that and then we had the challenge of taking hundreds of thousands of routers which weren't rdma capable and upgrading their software to the point that they could run uh protocols like priority flow control and give us the telemetry that we needed in order to run our network we were aided there by the fact that we used an open source operating system called sonic on many of the switches in our network even though the switches come from different manufacturers and have different asics in them because we're running the same software on them we were able to make sonic rdma ready and then all of our switches became rdma ready but still getting through those deployment challenges was actually quite interesting i'm going to change gears one more time and talk about another set of challenges that we saw let me actually roll back a little bit if you will give a dramatic oversimplification of the history of where cloud networking came from so cloud networking essentially started with this notion of software-defined networking the idea that we could suck all of the policy out of physical switches and co-located on some logically centralized controllers the idea of doing that is allow us to simplify the switches and have that centralized network intent control how data flow through the network well that idea of software-defined networking then got combined with this notion of network virtualization we realized that rather than having the controllers control physical switches in the network we should have them control virtual switches software running on each of the hosts in the network and that's where we should do all the sdn policy implementation apple's metering you know packet transformations to that idea of network virtualization we then uh added the concept of a very homogeneous extremely high scale extremely high performance cloud scale network typically built around a modified clo topology using things like ecmp to allow us to put essentially as many servers into a data center as that we had enough power to support by combining those three concepts together we essentially got to the point that we became freed from physical scale limits and those had always been the things that constrained the size of networks in the past but even after we put these three concepts together so that the physical scale limits weren't an issue now we're in a world where we hit the next set of scaling challenges and that's around the software so let's think a little bit about what that sdn control plane has to do using some statistics from azure we have over a hundred million sdn policy objects that are under management at all points in time we get over two and a half billion requests every day to read modify create those policy objects each of the customers hosting on azure expects to be able to make rapid changes to their deployments so for example each of those customers might expect to be able to create or destroy 20 000 vms in a minute each minute what that means is now building software a reliable distributed software control plane becomes a really interesting and hard challenge so for example one of the standard tricks we use in distributed systems to get more scale and availability is partitioning state we partition state across customers tenants we partition state across our failure domains however customers want a seamless update of all of their assets all at the same time now our customers have probably also distributed their their assets across multiple fault domains in order to get availability but again they want any change they want making policy to be seen in a consistent way across all of those that makes it very hard to use the standard trick of partitioning state one of the approaches we've used to solve this problem is building extremely high scale published subscribe systems for example things that can do 100 million notifications per minute over 100 million different partition keys another challenge we have in that control plane is making sure that the goal state is always correct and it's consistent the challenge though of course is that as customers are updating that goal state what sdn policies they want implemented what vms they have what ip addresses are assigned to those vms that involves complex multiple operation workflows and things can fail halfway through and we're frequently updating the code cloud never sleeps so we need ways of updating our control plane code even while these operations are in progress that means that we need to go and build systems for continually validating the repair of the goal state final challenge we're looking at is these policy scale challenges imagine a company out there with thousands of stores offices all over the place and each one of those has multiple vultural networks the company needs to control how the various assets they have for example point of sales terminals database servers communicate with each other and they need different peering arrangements for each of those kinds of virtual networks they need to be able to change deploy their change safely across the network essentially we've gotten to the point now that managing sdn policy is as complicated as managing network devices were back in the bad old days before we had sdn and so there are new research challenges to be faced in in solving the complexity of sdn policy one of the things we've been working on is azure virtual network manager to enable us to scale out that sdn policy introducing high level abstractions enforcing simple concepts like least privilege so that customers can be sure that their networks are safe and that only the communications they explicitly allowed are permitted and solving other challenges like how to combine security policies together so the outcome of adding multiple securities policies are predictable i'd like to thank you for for taking this tour with me through some of the scaling challenges we've seen in azure networking and i'm looking forward to listening to the other talks in this session thank you