Beyond the mega-data center: networking multi-data center regions (SIGCOMM 2020 Talk)

Key Takeaways

hi my name is voiceless jukich and i'll tell you about project iris project iris is a multi-year effort in microsoft research cambridge that explores small designs of regional cloud networks the project is motivated by rapid growth of cloud network traffic across data centers project iris is a part of a bigger office for the cloud team at microsoft research cambridge which explores disruptive technologies across the entire cloud stack at the beginning of cloud era data centers were built in remote areas that had cheap power connectivity and sufficiently large piece of land to accommodate mega data centers with tens of thousands of machines however finding such large plots of land in regions like europe or the far east is challenging moreover there is an increasing desire to minimize latency to customers by placing data centers close to large orbit regions which altifies the problem of finding land for mega data centers today cloud providers take a different approach instead of building one mega data center per cloud region they split it into multiple small facilities spread around the area and can connect them with a dedicated high bandwidth metric called data center interconnect or dci the map shows how the geography of such a region with four data centers might look like in the metropolitan area of lava each major cloud provider operates tens of regions around the globe with each region consisting of multiple smaller data centers connected by dci the impact of a shift to regional architecture is evident in the number of deployed 100g dwdm ports over time where growth has been exponential year over year it is important to know that the total number of pores deployed in dci is two orders of magnitude higher compared to number of ports in wide area networks this makes data center interconnect one of the most important pieces of the cloud puzzle so then how do we interconnect a few data centers with intensive kilometers from each other using the available fiber in the region we can interconnect data centers in various ways starting from a centralized topology with only one switching point for the entire region over semi-distributed with several switching points to fully distribute designs where each data center is directly connected to every other data center in the region we perform a detailed analysis of today's design space across a number of dimensions based on data from asia regions and testbed experiments as values show dci design involves complex tradeoff across a number of metrics certain solutions reduce latency at the same time increase deployment flexibility which is essential in urban areas although they look like an ideal design points they have a serious problem the cost of these solutions is so high that they become practically invisible because of these insights we propose a new all optical dci architecture that has positive impact on the cost across the entire design space and practically unlocks previously expensive solutions let's now look at dci problem in more detail we start with the impulse to design process first we need a map of available fiber in the area here you can see a map of dark fiber in london operated by only one fiber provider where each blue line represents a fiber top metric areas are often full of low cost fiber a single fiber dock commonly has hundreds of fibers that can be used for building data center interconnects second we need location and the number of data centers typically each region has less than 20 data centers also it is important to know the capacity that must be supported by the network data center number location and capacity are determined using variety of business objectives which are outside the scope of this work since we don't control them finally we need to look the location of so-called fiber hubs these are very small facilities that are cross points of fiber cables and can be used to host switching equipment throughout this talk i'll use this simple example which has four data centers marked in red blue lines represent fiber dots while brown points are fiber cuts using the inputs above we want to design a network that achieves a certain set of goals and operational constraints first we want to provide a latency sla to assure that the latency between a pair of data centers will not exceed a certain threshold we have to bound the maximum distance between data centers in the agreement with dci practices we bound distance to 120 kilometers also a pair of data centers should communicate over the shortest path available this requirement is motivated by minimizing application latency second data centering connect interconnect needs to be able to satisfy any traffic pattern that does not violate per dc capacity regional data centers should be able to satisfy the same traffic characteristics as if they were in a single mega data center in third we want to satisfy previous two requirements under a set of failure scenarios as a running example we consider up to two edge failures in the network topology which is aligned with today's industry practices now since we know what are our goals let's talk about design process itself what does it mean to design a data center interconnect first we need to decide which fiber dots to use and how much capacity in terms of number of fibers to reserve in each one of them and second we need to decide which fiber huts we want to use to place network equipment for switching and routing let's first examine a candidate dci architecture which uses a centralized hub all data centers connect to the hub and all dci traffic goes through the hub designing and provisioning a hub is a well understood problem in networking in practice though two hubs are necessary for redundancy all the simple decentralized architecture has several problems first it can cause a latency inflation we demonstrate that on a real topology with two data centers and two hubs two data centers at the top while two hubs are at the bottom black lines represent available fiber ducts in the region when data centers want to communicate they have to go through the hub note that although there is a direct short link between them the centralized architecture does not exploit that link this limitation increases latency by six times in our analysis we show that at least twenty percent of cases we consider we observe at least 2x latency inflation caused by a centralized design another problem is the centralized approach limits the data set replacement flexibility due to latency constraint each data center must be within a certain distance r from a hub as we show here since the latency sla must hold even under failures each data center must be placed within distance r from both hubs this means that the real service area where we can place data centers is within the intersection of these two circles placing hubs closer to each other increases service area but may also cause latency inflation as we demonstrated previously also hub schools to each other have higher probability of failing together this exposes an interesting trait of decentralized topologies we have to pick between reliability and latency on one side and service area and sitting flexibility on the other another approach to building dci would be to use direct links between each pair of data centers all the expensive and complex this approach has certain benefits first the latency between each pair of data centers is optimal because a pair of data centers is always able to use the shortest path possible when it comes to the size of surface area distributed topology allows greater flexibility here we show an example of surface area for six data centers marked in yellow and two hubs marking blue in london for a centralized topology service area is a result of intersection of two circles around two hubs if you remove the hubs and make the topology distributed the service area increases significantly this happens because data centers are now not limited by the hubs with small radius of 60 kilometers but rather each data center can talk to any other data centers within the radius of 120 kilometers this is not the isolated case we calculate service area for centralized and distributed topology across many real cloud regions and show that distributed approach can bring up to five times more area compared to a centralized approach which provides a huge advantage in planning from cloud regions in densely populated cities although distributed approach has substantial benefits in terms of latency and the size of service area it increases the complexity of the network let me illustrate that using an example here we have a centralized topology where each data center requires ports to connect to the hub if a new data center is added only that data center and the hub must be changed while other data centers remain untouched however in a distributed approach each data center now requires three times ports more generally for a region with n data centers the region can be implemented with order of n square ports while centralized topology requires only order of fan reports this directly translates to more management complexity and higher cost also if a data center is added to the topology there are many places in network that require changes in capacity provisioning comparing just the poor cost for a region with 16 data centers a distributed approach is seven times more expensive compared to a centralized topology due to substantial difference in total number of ports they required looking at the entire design space by moving from a centralized to distributive topology the latency and seating flexibility improve but at the same time the cost increases drastically additionally as we move toward more distributed topologies the complexity of the network increases as there are many more components to be managed from the operational perspective centralized topology has the most desired properties it is simple to manage and extend practically the increasing cost and complexity restrict design choices that cloud providers have today toward more centralized approaches this motivated us to propose iris a new all optical data center interconnect architecture that reduces both cost and complexity and allows the entire design space for practical purposes so how does iris work to understand how iris reduces cost and complexity we first have to take a look at the main inefficiency of today's dci deployments number of ports and receivers they require since the traffic is switched electrically typically in today's dci every time a fiber arrives at a switching point it must be terminated by 40 transceivers and ports one per color also leaving the switching point requires another 40 transceivers this quickly inflates the cost especially in a distributed topology in our example if each data center operates only a single fiber we would need to deploy a total of 320 expensive e-network receivers as well as ports instead of frequent signal conversion and electrical switching iris relies on switching the traffic entirely in the optical domain which should reduce the total number of components necessary since there are no e-network receivers in this particular example iris requires only 8 optical ports instead of 320 and no e-network receivers to understand what the transition from electrical to optical switching practically means let me start by taking a more detailed look at today's dci links multiple transceivers at dci1 are combined into a single fiber and sent to switching point where the traffic is converted back to electrical signal switched and then again converted to the optical domain where it's finally forwarded to the destination as the optical signal travels through fiber and other optical components it loses power so we need two amplifiers per point to point link one amplifier to compensate the loss because by packing multiple cores into single fiber and another amplifier to compensate the power loss caused by long distance fiber now ideally we would like to replace only electrical switching part with optical without touching other components in the system unfortunately this is not possible optical switching introduces additional challenges that must be considered these challenges need to be solved taking into account specification of dci transceivers and other physical layer components in this context a signal should arrive at a receiver with sufficient sufficient signal power also signal needs sufficiently high optical signal to noise ratio and third optical control plane can become quite complex so we want to simplify it by minimizing number of devices that require reconfiguration first optical amplifiers introduce noise to the signal more amplifiers means more noise the cell can be decoded only with sufficiently high optical signal to noise ratio or osnr as we experimentally show in the paper without signal regeneration os in our budget allows us to safely use only one e-network quantifier besides worrying about signal quality we have to guarantee that signal arrives at a receiver with sufficient amount of power this is why we have to carefully place the one e network amplifier in the paper we provide an algorithm for amplifier placement and guarantees meeting all optical constraints when it comes to switching technology we can take two approaches wavelength switching that allows us to switch at the level of each individual wavelength and cores create fiber switching that can move only the entire fiber although wavelength switching provides finer switching generality we can only have one wavelength switching point end to end due to osnr and power constraints this limited form of wavelength switching introduces substantial complexity and brings marginal cost savings as we show in the paper this is why artists relies completely on simple fiber switching technology however fiber switching does not come for free it actually requires additional network capacity consider a traditional link with electrical packet switching that goes from data center 1 and can receive fibre's worth of traffic to any other destination in the region in the paper we show that if we would like to implement an equivalent link using fiber switching we would need c plus n minus 1 physical fibers to guarantee non-blocking network where n is number of destinations this results in order of n square additional fibers for a distributed topology although not perfect we are ready to take this overhead because regional fiber is cheap relative to transceivers and the total number of destinations is commonly less than 20. also note that the fiber overhead does not require laying new fiber ducts in the metro area existing fiber ducts usually carry hundreds of fibers with many of them being currently unused the third challenge we have to consider in the optical domain is what happens to a system when reconfiguration is necessary due to changes in traffic demand we illustrate the major problems on simple topology with three data centers initially data center one sends only two wavelengths red and green to data center two assume that the traffic demand changes there is no traffic between data center one and data center two but there are two new wavelengths green and blue coming from data center 3. first the receiver at data center 2 will notice much lower signal power due to longer distance than new c not traverses and second the output of the amplifier in the middle depends on the colors it amplifies a different set of colors can lead to a completely different amplification result dealing with optical amplification of a signal that varies in power and the colors it carries has been a serious challenge in noctis community with no easy solution this is why we decided to avoid online adjustment of power levels by first introducing power limiters in front of amplifiers that prevent changes in the single power due to changes in distance and signal sources second we introduce noise sources they fill partially filled fiber with valence that are missing so optical amplifiers always see the same set of colors noise sources together with fiber switching and power limiters guarantee that no online adjustments or power level is necessary even under traffic changes we implemented the full physical layer architecture in so to say region in iraq using all components of a dci architecture including transceivers amplifiers tens of kilometers of fiber as well as wavelet and fiber switching equipment this resulted in a small scale region with three data centers that reassemble the microsoft research cambridge iris architecture does not require active management of power levels significantly lower in the bar for deployment and our analysis on the real equipment shows that this is feasible with truly minimal performance impact since we show that rs can be practically implemented i would like to quantify its benefits compared to traditional electrical links two identical topologies one implemented using electrical links and another using iris by design satisfy operational constraints in terms of non-blocking switching resilience and latency performance however the overall cost of these two systems is very different we tested the cost across 10 real fiber maps and different placement of data centers in a distributed topology and using real cloud provider component prices amortized over time we showed that iris reduces the cost by 4 to 12 times compared to electrical switching also we evaluate the complexity of two approaches in terms of total number of in-network ports and show that rx reduces complexity by an order of magnitude before i conclude my talk i would like to highlight several interesting pieces of our work that are admitted due to time constraints but you can find them in a paper detailed analysis of latency and service area in regional networks based on real cloud regions algorithms for capacity provisioning under failures and amplifier placement that minimizes the overall cost large-scale simulations and analysis of dynamic network behavior and reconfiguration many physical layer experiments and finally we provide a whole new design and analysis of an architecture that combines valence switching with iris to minimize the fiber overhead that iris has though that to truly unlock benefits of the whole design space we had to jointly solve a wide spectrum of problems across technologies and network layers these include challenges like topology designed to meet slas capacity to provide non-blocking connectivity and resilience to ensure that slas are preserved even under failures all that has been done while respecting many physical layer implement limitations as we previously discussed this all resulted in iris and all optical dci architecture that unlocks advantages of distributed topologies while lowering the cost and complexity it provides benefits across the entire design space iris was possible thanks to multiple multidisciplinary optics for the cloud team at microsoft research cambridge and insights across the entire cloud stack thank you all for watching