Lecture 7: Recurrences

Key Takeaways

all right uh let's get into it um last lecture of material before the quiz so let's make sure we make the most of it today we're talking about recurrences uh recurrence is a fancy word for saying sequence described by induction uh so it's a sequence um defined inductively and we've seen multiple examples a great example is the Fibonacci numbers um are there two B's uh Fibonacci let's pretend I spelled that right no it's two B's and one n isn't it are there two C's I don't know no one's no one's helping me um doesn't matter um defined by F0 is z fub1 is 1 and the important part is that FN is defined as FN minus1 plus FN minus 2 when n is at least two and this is the part where we're describing the sequence inductively once you know the first two terms the next term is the sum of those so F2 is 0 + 1 is 2 F3 is 1 + 2 is 3 then 2 + 3 is 5 3 + 5 is 8 then 13 21 34 and so on um it's just a sequence defined inductor L that's the idea of a recurrence and these show up all the time when analyzing algorithms for many of the same reasons that summations show up when analyzing algorithms and we need a lot of the same things out of them if we end up with a an algorithm whose runtime we can describe as the Fibonacci numbers or as a similar recurrence well that's not as useful as it could be um a closed form would be a lot more more helpful right um if we could have it uh if we could find it and indeed Fibonacci numbers do have a closed form the nth Fibonacci number is 1/ sarek 5 Time 1 + < TK 5 / 2 to the N - 1 - < TK 5/ 22 to the n and there's a closed form you just take two exponentials um subtract them divide by root 5 and you get the nth Fibonacci number exactly I think this is a really cool formula also non VI how how do we possibly go from this to that um well we have some techniques that might work sometimes and um other times we won't um but certainly once we know this formula if if we believe this formula or want to check whether it's true now that we know the formula we can just try the induction and see if the induction works out um in this case it will work out and then you'll not only know that this is the right formula but you'll have a proof for it so just like with summations if you know a closed form um then usually it's easy to prove it by induction so finding the closed form is the hard part proving it is the easy part so this guess and check guess and check technique that we used for sums works just as well for recurrences all right um now recurrences very often show up um especially when analyzing recursive algorithms so let's talk about recursive algorithms uh and show some examples so let's start with a very Classic Fun example called The Towers of Hanoi so this this comes with its own myth um it was described by Edward lucah in 1883 he said there was a temple in anoi that has three large pillars um I have I have some helpers today acting as my pillars we've got Ptown wire and tooro um and on the leftmost pillar are 64 golden discs stacked on top of each other getting bigger as you go down uh and the monks there are moving the discs around one at a time um but according to a couple rules you're only allowed to move one disc at a time from the top to some other Peg and you're never allowed to put a big box on top of a small I mean big Golden Disc on top of a small Golden Disc um those are the rules one one peg at a time so I can't pick these both up and move them elsewhere um and always small on top of big never the opposite the goal is if all the discs start over here on the left Peg we want to get them all to a different Peg we want to move them all to the right Peg um by moving one disc at a time so let me write that down quickly so we have let's say n diss on left Peg we have two other pegs and we're going to move one at a time um always keeping uh smaller discs on top of bigger ones goal all discs on the right Peg and as the myth goes when the monks finish when they get all 64 discs over to the the other Peg the universe ends I don't know why that's that's just the story maybe there's a um a weight sensitive um detonation switch over there and you have to get all that weight on there and then then the universe blows up it's a myth you don't need actual answers here um but that's the game uh and the question is how many moves will it take is it even possible how long does the universe have to live do we have a year do we have a a century um let's see if we can figure it out let's see if we can find an algorithm to get all of the the discs from one side to the other are we clear on the game nice let's start small I always recommend starting small what happens if we have just one disc and we need to move it from Ptown to tooro how are we going to do that is that a hand so how how do I move a disc from here to here according to the rules of the game yes you just do it yeah we're allowed to move one disc at a time and there aren't any bigger or smaller discs to get in the way so you just do it in this case um so if we're trying to move um just dis number one from let's say Peg a to Peg C we'll call them the pegs ABC well you do it just by doing it you move you move a disc uh you move dis one from a to c great okay um what if we have two discs here we go how am I going to get both of these from there to there who can help me out yes please small one in the middle second one goes over here small one on the big one absolutely perfect so if we're trying to move discs one and two the two smallest discs again from Peg a to Peg c um as your colleague pointed out um let's move first let's move dis one from A to B then we'll move dis 2 from a to c then we'll move dis one from B to C just as we acted out over here next case three boxes now this there we go um this might take a little little bit of planning ahead um and the thing to remember is that at some point we want to be able to make this move where we take the biggest box from the left to the right and so to do that we need the other boxes here in the middle so somewhere along the way we're going to need to go from here to here of course that's not a valid move we can't move two boxes at once who can help me out how are we going to do this first part of the algorithm please small box to the middle we're trying to move move these two to the middle so here to here is our goal small box to the right medium box to the middle small box on top of it perfect does that look familiar to anyone that's kind of exactly what we just described for moving boxes One and Two from one peg to another instead of moving from the left Peg to the right Peg we're moving from the left Peg to the middle Peg but that's the only thing that's changing we can use the same algorithm to move these two boxes from here to here in three moves now we can do this and now we can do the same algorithm again to move these two boxes from here to here and if we think about what that algorithm says it says one 2 3 let's write that one down as well so if our goal is to move the interval 1 through three so boxes 1 two and three from Peg a to Peg C well then the first thing we're going to do is move boxes one and two from A to B then we're going to move box three on its own from a to c then boxes one and two from B to C and here when I'm saying move two boxes at a time from one peg to another I'm really invoking this strategy we already know how to do it with two boxes so let's use that as the first part of our algorithm then make one move then use that again does that make sense let's do one more example together okay great we have four discs we need to move them from the left to the right um and we're going to follow hey everyone we're going to follow the same idea um somewhere along the way we're going to need to get to this position to this position so that we can move the bigger box over here and then we're going to need to again move these three boxes together somehow with some sequence of moves to finish the job and I claim we have all the tools we need to do exactly that so step one in fact let's write this down first if we're trying to move the interval from one to four so boxes 1 2 3 and four from a to c let's follow the same strategy here let's put the the top three boxes let's move those from A to B somehow then we'll move box four on its own from a to c then we'll move boxes 1 through three from B to C and here where I'm saying move three boxes from one peg to another we have an algorithm for that it's this algorithm yeah so let's let's see what this looks like in practice so first things first we're trying to move these top three boxes from here to here um well we have a to do that that's 1 2 3 4 5 six seven if we follow the algorithm that's listed there those are the seven moves it says to do and I want to point out one important thing we put this one over here back when we were just working with three boxes we were assuming that all of the pegs were empty that there was nothing else to worry about other than those three boxes but now there's a fourth box in in the way so can we really run our algorithm for the top three boxes even when there's a fourth box and yeah we can thankfully because the fourth box will always be underneath the fourth box doesn't affect the movement of one through three in fact any larger boxes don't affect the movement of boxes 1 through three so we can use our algorithm for three boxes and just temporarily ignore the fact that there's a fourth box it doesn't matter and that's how we were able to get here then we're going to do this and then again we're going to follow our our three box strategy to get these three over to here so that looks like one 2 3 4 5 6 seven and we've done four boxes yes great question how are we able to claim that this sequence of moves especially with more and more boxes or discs is the optimal algorithm we've we've started describing an algorithm to do it but is it the best uh that's a great question um turns out it is the best um we're not going to prove it right now but it's not much harder than what we've already done um the basic idea is at some point you have to move the smallest at the largest box so at some point you have to get to either this position or this position and that and we know how many steps that takes assuming we're doing some proof by induction or something so we know we have to spend some number of steps for the um for n minus one boxes then we have to make a move and then we have to put these back on top which again takes a land slide so with a with an induction very similar to that you can prove that in fact this is the optimal algorithm um okay but how many moves does it take let's see if we can figure that out um so let's let um T of n be the number of moves with n Diss and we we want to figure out T of n um somehow so here T of one if we're just moving one disc we make one move so T of one is one t of two well T of two is written here it's one two three moves if we're moving two discs we go one two three uh three moves how about t of three uh well it says do the two disk algorithm so that's T of two then do a single move then do the two disk algorithm again so that's plus T of2 again so that's 3 + 1 + 3 is 7 likewise T of four is well you do the three disk algorithm that's T of three we do one move then we do the three dis algorithm again that's another T of three so we're looking at two t of 3 + 1 which is 14 15 and now we've exactly described the number of moves that this algorithm uh takes and just to really be clear about it if we're trying to move diss one through n so n discs together uh from Peg a to Peg C we're going to first move uh the first n minus one discs from A to B then we're going to move uh disk n on its own from a to c then um the remaining n minus one diss from B to C exactly the same strategy we've just we've just described um and this is going to be our general strategy to move n diss we do our n minus one dis algorithm then a move then our n minus one dis algorithm again so now we've described an algorithm by induction and secretly that's all recursion is we're just inducting um but in general the number of moves we take is well however many moves we needed to take for the N minus one dis algorithm + one plus TN minus one again so it's 2 T of n minus1 plus one and this is our sequence this is our recurrence and we're getting to the place where these are in the way so let's move those success okay we've described our algorithm we've described its runtime but like I said earlier closed form would be so much nicer right um I want to know how long the universe takes with 64 discs I don't want to have to compute 64 terms of my sequence so what does this look like well let's see um we have our our numbers 1 3 7 15 2 * that + 1 is 31 then 63 then 127 uh is anyone starting to see a pattern uh yes uh we have a guess T of n is 2 to the N minus 1 this looks suspiciously like 2 4 8 16 32 64 128 um and especially since we really are doubling every time it seems likely that it might be related to powers of two somehow well the good thing about having a guess is that we know what to do with it we check it we try to use it to prove this formula by induction and if it works we have the right formula turns out it works you can prove it with a a pretty straightforward induction um and turns out this is indeed the correct formula so this is a clear example of uh what we said about guess and check if you can guess the formula you check it with induction but we had to recognize that pattern to be able to have something to check okay but finally we can answer our question how long does the universe have left um we it's 64 discs so two to T of 64 that's 2 the 64 minus one steps let's assume these uh these monks are really strong and they can move one disc per second um turns out if I flip to the correct page in my notes um that many seconds is about um a half trillion years Universe has got some time left and I find it surprising that with just 64 discs you can get all the way up into the trillions of years uh that's very quick growth um all right any questions about towers of Hanoi awesome I thought the camera in the back was a hand for a second I apologize for that um let's move on to our next recursive algorithm um I promised last time that we would talk about sorting so let's let's talk about sorting uh and before we get to merge sort which is um the the fast algorithm we're going to talk about today let's talk about a simpler algorithm called selection sort um the problem is you have a list of n numbers um and you want to arrange them in increasing order so your list is in whatever order it it's in and you want to figure out the correct increasing order so put them in sorted order um and the idea for selection sort um is a very simple one find the smallest pull it out repeat until done so let's see this in action and I have some helpers today I'm going to need a few more volunteers um thankfully I brought some nice all right so we have a bunch of helper friends and we need to somehow put them in sorted order uh we're going by height here um and for selection sort um we said find the smallest well clearly that's Ptown here um pull them out and then repeat okay who's the smallest left well that's going to be wire who's the smallest left um well it's maybe a little less clear but let's talk more precisely about how we're going to find the smallest every time and we can do this with just a just a bunch of comparisons so let's compare the first two uh between Ralph and wire who's shorter Wire yeah okay so wire is the smallest one we've seen so far between wire and Ptown who's shorter P toown between these two who's shorter still p toown still P toown still P toown still P toown so we got through the end of the list Ptown is the smallest one we've seen so Ptown is our smallest softest roundest boy um and that is how we are going to find the smallest we can do that with just n minus one comparisons we compared the first two then whoever won that we compared to the third then compared that to the fourth then the fifth and so on so n minus one comparisons and then you pull them out and we're going to do it again so now with n minus two comparisons because we have one fewer animal up front we're going to find that wire is the smallest then we're going to find I think it's bubs is the smallest um then todoro then Ralph then Coupe get out of here um then butternut and now we have our sorted list how many comparisons did we have to do to implement that well we've already said that we did n - one comparisons in the first round and then the second round we had n min-2 comparisons the third round n minus three comparisons down to the last round where we had just two animals left we need one comparison to figure out who's shorter and then the last one is just the last one we don't need more comparisons for that um and we've seen this sum before this is n minus1 * n divided by two comparisons so this is um n^2 over two plus change comparisons for selection sort this is not the fast sort that was promised this is still quadratic runtime um it is the easiest thing like when you have a small number of things in front of you and you just sorry for calling you things small number of animals in front of you uh and you just want to sort them like it's very easy to just scan through find the smallest find the smallest that's what I do by hand but for big um for big lists for lots and lots of numbers this is not the way you want to go but um a much better algorithm not the only better algorithm but a much better algorithm is pretty similar to this um and to talk about it first let's talk about the idea of a merch the idea is we have two sort of lists A and B and we want um to combine them into a single sorted list um as an example um so maybe we have these and you know that goes over there um there toor there we go so for example if we have this sorted list which is already in increasing order and this separate sorted list that's already in increasing order let's try to combine them into a single sorted list and let's just do something like selection sort let's find the smallest and just keep finding the smallest until we have our answer so who's the smallest between these two lists where do we have to look well I I saw I saw this was that was that an indication all right so the beginning of here or the beginning of here yeah the smallest on the table is either going to be the smallest on this side or the smallest on this side and we know where the smallest is because the lists are sorted the smallest on this side is wire the smallest on this side is Ptown we just have to look at the first from each of these two sorted lists and with just one comparison we know who the shortest is overall still Ptown um and then we can continue um first over here compared to the first over here wire versus bubs I think I think wire is smaller um now it's bubs versus todoro um and I think bubs is smaller this time now it's Coupe versus tooro tooro uh Coupe versus Ralph Ralph is shorter um and now we get to an interesting point where there is no first over here anymore the side of the table is empty um well that just means that we're done we know this side is already sorted and bigger than everything else we've seen so we don't have to do any more comparisons we just pull them back and we're done so that's the idea of merging two sorted lists the algorithm says we compare smallest in each list um pull out the smallest continue until one list becomes empty and we put the other list at the end right so we got it down to where this list was empty this list had two animals left um and we didn't have to do any more work we just we're just done with those is this algorithm clear does it make sense why it works how many comparisons does it take well each round every time we identify um the next smallest animal we're doing a single comparison yeah in the worst case we're going to have to do that for all of the first n minus one animals um like if it comes down to one on the left versus one on the right then everyone needed a comparison we only got one freebie at the end so worst case um this takes less than or equal to size of a plus size of B minus one comparisons sound good awesome so now that we have this merge algorithm let's talk about merge source so you're given a list with n elements uh and we want to sort it um this list doesn't have any promises it it's not like half sorted or anything like that it's just some random list of random data we want to put it in sorted order um and by the way let's assume for now that n is a power of two so we can just keep cutting in half uh and everything works out nicely we'll remove that assumption later but for now we'll just solve this for lists whose lengths are powers of two uh so uh the first step we are going to sort the first and oh sorry I forgot an important step if n equals 1 done if you're sorting a list with just a single element there's nothing to do your list is already sorted so we'll call it done otherwise we're going to sort the first n/2 elements somehow we'll we'll get them sorted then we will sort the last n /2 elements somehow and then you merge them together so as long as we're somehow able to sort the first half and somehow able to sort the second half the merge step will take care of the rest does this strategy make sense cool and you might ask well how are we going to sort the first the first half and the last half maybe we'll use selection sort turns out if you do that you're still going to get quadratic runtime that's not not the best plan um the strategy I'm going to propose and the way that merge sort is described we're going to sort the first n/2 elements using merge sort and we're going to sort the last two last n/2 elements using merge sort okay I claim I have written an algorithm here though it might not be obvious recursion in general is one of those bogeyman topics it's seen is like the bane of all computer science students um woe unto me the end is nigh it's the hardest hardest ever uh topic uh and I am here to tell you it's not some new complicated scary topic it's an old scary top no it's an old familiar topic um it's just induction I claim everything we're doing here is just going to be induction and let's talk about what we mean by that um so let's run through merge sword and see what it looks like so we have our scrambled list um which is a power of two which is not a power of two which is a power of two oops um all right last helper for the day thank you Charlie um now let's scramble this up and see if we can sort them together so here's Butternut over here all right so let's run through our algorithm and see what it tells us to do so merge sort on eight elements so we're calling merge sort on eight elements I'm I'm going to abbreviate over here just to keep track of where we are um and it says the first thing to do is merge sort the first four elements then merge sort the last four elements then merge them together so we're going to merge sort 1 through 4 and then we're going to merge sort 5 through 8 and then we're going to merge right that's what the algorithm tells us to do so it tells us take these four I did not do a good job scrambling these did I let's let's do a little more um okay we'll pretend we did that um so it tells us to merge sort the first four elements then merge sort the last four elements then merge them together okay well here we're merge sorting the first four elements what does our algorithm tell us to do when we're merge sorting four elements well it says merge sort elements one through two then merge sort elements 3 through four then merge them together yeah I'm just reading the code here and that's what it tells me to do when I call merge sort on these four elements so it tells me to call merge sort on these two then call merge sort on these two then merge these together into a single list a single sorted list of four elements yeah okay well what does merge sort with two elements mean okay um sort the first one element sort the next one element then merge merge sort on two elements well sorting one and one on their own there's nothing to do a sort a list with one element is already sorted so all I have to do is the merge merge this list with that list and who's who's shorter Ptown than bubs okay now this list is sorted merge sort on these two elements next um here's our you're kind of heavy um this one over here this one over here we get to do our merge so who's shorter it's Coupe then it's Charlie so where are we again we've done this and this now we have to merge them I'm just following what the code says to do um we have to merge this list with this list okay between these two who's shorter Ptown between these two who's shorter bubs now one list is empty so the other list just Falls in place and now we have successfully called merge sort on these four elements let's do the same on the other side so split into two and two no better not um split into one and one merge them together toor Row first split into one and one then merge them back together wire then Ralph now we have two versus two that were merging we have wire versus todoro that's wire we have todoro versus Ralph that's todoro oops there we go then Ralph versus Butternut Ralph then butternut and now we have sorted these next four elements as well and the last thing we're supposed to do is merge these four with those four okay we've got these two it's this we've got these two it's this these two I think it's bubs these two todoro these two what was it again I think it's Ralph then these two is Coupe then Butternut then Charlie and so we just ran through merge sort on these eight elements just by following this code and it kind of worked right we we got our answer in the end and it worked but how can we be confident that it's going to work more than that how can we be confident that we've even described an algorithm because like how can we describe the merge sort algorithm using merge sort before we finish describing merge sort how can we know that this recursive call to merge sort correctly sorts our list before we've proven that merge sort correctly sorts your lists this this is the kind of circular thinking that makes recursion hard to think about but thankfully we don't have to think about it in that way we can think about it as just induction we're inducting on the size of the list that we're sorting merge sort on one element we know what it does and We Know It Works it does nothing and it's correct if there if we're sorting one element we're good we're good merge sort on two elements um says we we're just doing that one comparison like we saw this also works merge sort on four elements to describe merge sword on four elements we call merge sort with two elements then merge sort with two elements then merge them together but by the time we're here we already know what it means to merge sort two elements and we know that it does the job correctly so these recursive calls don't think of them as calling back to merge sort which we haven't finished describing yet think of them as calling a previous incarnation of merge sort we're calling a smaller size which we've already constructed because we're building them up inductively starting from the smaller sizes and working bigger so we know that merge sort 2 exists and is correct and we use that to prove that merge sort 4 exists and is correct we use that to prove that merge sort 8 which is using merge sort 4 both its description and its correctness to make sure that merge sort 8 works and is correct once we have once we know how to merge sort eight elements we can use that to know how to merge sort 16 elements we can use that to merge sort 32 elements so all this circular reasoning we can break that cycle by remembering that the recursive calls are actually your inductive hypothesis we're allowed to assume that that it works for eight elements when we're constructing and proving it for 16 elements and so on and so forth does that idea make sense awesome all right um it can be helpful to see it all at once and not with with cute fuzzy helper friends uh so here's a a quick animation that's just running through the whole algorithm um and every time has an unsorted list it's breaking it into two halves and then by the time they come back sorted it's doing the merge uh so we're about to merge 5 seven with one two looks like one is smaller than five then two then five and seven have no friends so they go back in um now we're going to call merge sword on the right half um here we're merging one and one to get two um now we're splitting and now we're merging back uh 0 and six now let's merge 34 with 06 looks like 01 then three then four is the winner and then six and finally we merge the two sorted Hales with our familiar merge algorithm again uh next three beats five four beats five five beats six then six then seven uh and that is a quick view of merge sort with just eight elements are there any questions on how this algorithm works all right still restricting to just powers of two for now um well how long does it take we have an algorithm I want a runtime uh and just like before we can describe our runtime with a recurrence um let's say let's call it like M of n which is this is the um an upper bound on the number of comparisons when merge sord n elements and we're trying to figure out M of n for now just when N is a power of two um and we have our algorithm it's up here um the runtime for M of n well we're calling the algorithm for n /2 and we're calling it twice so 2 * m of N /2 and then we're merging n/2 elements with n/2 elements and that merge takes at most nus one more comparisons and so M which is described by this recurrence is an upper bound on the number of comparisons we'll need does this make sense how we were able to translate this recursive algorithm into this recurrence um for a bound on the runtime this isn't the exact runtime because like we said um merge sometimes takes fewer comparisons um if one of the lists is entirely smaller than the other lists then you only need n over two comparisons and then the other list falls into place for free so it's going to be somewhere between n/2 comparisons and N minus one comparisons worst case it's going to be n minus one and since we're looking for an upper bound here we'll call it n minus one okay well we have a recurrence um we can compute some terms and aim for a closed form see if we can get an answer there uh and let me write down some terms for us here's n uh here's selection sort uh and here's merge sort just for comparison uh n is 1 2 4 8 16 and 32 that should be enough to find a pattern right so for selection sort that was the slow quadratic one um we had zero 1 6 uh 28 120 and 496 and we already know the formula for this that's uh n^2 - n / 2 uh for merge sort we get 0 1 5 17 49 129 and we don't know this formula yet gratifying to see that it's smaller than selection sort so this looks like an improvement on our algorithm uh can everyone see or is Charlie in the way sorry about this but looking at that I I don't really see the pattern I can't think of a formula that gets me those numbers just by staring at them which brings us to our other technique for trying to find a closed form when we don't know one yet um just like with sums we had the perturbation method um when we were trying to figure out a closed form well here with recurrences uh we have what we call plug and chug not to be confused with guess and check whoever named these I don't like them the names not the people they're fine I assume um we have the plug and chug technique which looks like the following so roughly speaking um substitute the recurrence into itself so we get um expanded and expanded versions of the formula and look for a pattern in that expansion let's see if we can do that with merge sort so we have M of n we know I'm going to put the N minus one first this is n-1 + 2 * m of n /2 right that's just our our recurrence exactly as it was um described before but this m of n /2 m still satisfies our recurrence so let's apply this recurrence to n/2 and replace this term with the recurrence version of that so m / n is going to equal nus1 + 2 * n / 2 -1 + 2 * m of n over4 so I applied our recurrence to the input of n/2 and substituted that in here are we clear on how I got from here to here this is the key step um we can expand this out a little bit um so this is n minus one this is 2 * that is going to be n minus 2 and then we have a 2 and a 2 + 4 or M of n / 4 all right so we expanded it out one step and now we have this recurrence which still describes the same sequence um and we can do it again let's do it again this is n -1 + N - 2 + 4 * all right n over 4 -1 + 2 * m of n / 8 pretty sure I did that right we can expand that out a little bit uh so M of n is still equal to nus1 plus nus 2 plus and now we have four times this which is going to give us n minus 4 plus there's a four and a two we have 8 times M of n over 8 all right and if we keep doing this we can we can guess at the pattern we're going to see when we expand this further and further we can guess that M of n is going to look like nus1 plus nus 2 plus N - 4 + N - 8 plus N - 2 to the um T -1 + 2 the T * M of n / 2 T if we expand this many times it looks like we're going to end up with this this was just me sort of guessing at the formula we're going to get but informed guessing by how this keeps expanding um and remember we're already assuming that N is a power of two we're assuming that n is 2 to the K so what happens if we choose t = k then M of n would equal n -1 + N - 2 um plus N - 4 plus N - 2 to the K-1 plus 2 the k m of n/ 2 the K okay but 2 to the K is n and n / 2 to the K is 1 so this term here is n * m of 1 yeah all right let me make sure I'm still matching my algebra and then from here we have we have a sum to deal with um let's take a look at all of these n terms how many are there well it's 2 to the 0 up to 2 the K minus one so there should be K of them yes instead of what 2 to the K so 2 to the K is n uh we were assuming that N is a power of two and K is that exponent um so um these two are the same term good question so let's see what we got how many n's did we have we have K of them this is K * n now let's deal with these parts that are being subtracted minus 1 + 2 + 4 + up to 2 the kus1 and then this term here what was M of one again yeah I I see someone going like this excellent zero yeah there's nothing to do when when we have just one element M of one is zero so that whole term in fact disappears and we're just left with this which has knen and a geometric series and we know formulas for geometric series I'll spare us the the details uh this is going to be equal to um KN n minus 2 to the K minus one and if we remember what K is K is log 2 of n this is n * log 2 of n that's this term and then this 2 to the K is just n so minus n + one is our formula and we got that not by looking at the numbers but by expanding out the recurrence and looking for a pattern there of course this is still just a guess we didn't really explain why this thing is true we would maybe want to prove that by induction but easier option once we have our actual guess let's just prove this by induction once we have our guess we can check it and turns out this is the correct formula there's your check questions on that plug-and chug method really useful method doesn't always bear fruit um but often does and it's a a nice tool to have in your back pocket one bit of advice when using plug and chug by the way is notice when I had this n minus one and N minus 2 I didn't combine them I could have written that as 2 N minus 3 and then the next one would have been what 3 n minus 7 um but it's maybe the pattern is less clear if we did that I tried to keep terms from the different levels of recursion separate from each other so I can keep track of what's coming from each stage um so when you're doing plug and trug try not to simplify everything at the beginning uh it might help you see the pattern more readily all right back to merge sort finally what if n isn't a power of two I claim the algorithm works pretty much as described we don't really need to change it all that much um at least we don't have to change the idea just the measurements uh so we're going to sort the first n /2 floor elements we're going to sort the last ceiling of n/2 elements and then merge them together um excuse me so this notation floor notation this means n/ two rounded down to the nearest integer so if n was 11 we would be sorting the first five elements and then sorting the last six elements and we again have two sorted lists now with different sizes but that doesn't matter merge doesn't care what size the lists are we can merge them together and this is our actual implementation of merge sort no matter what the size is uh and we can similarly bound its runtime uh now M of n the run the number of comparisons when merge sorting n elements uh or at least an upper bound on the number of comparisons um well now we're recursively calling floor of n/2 and ceiling of N /2 and then still doing a merge with n total elements um we can be conservative and say the number of comparisons is at most two times the amount of comparisons it takes to sort the bigger of the two halves um if the other half is smaller then presumably that's even fewer comparisons um so this this recurrence and then still plus n minus1 for the merge at most so this is a conservative measurement of how many comparisons we need in the worst case for n elements are we okay with how we got here nice and now we could try the same thing we could try to like plug and chug or something or guess and check and see if we can get a closed form for this recurrence here I don't want to um especially the the floors and SE or the ceilings rather uh they kind of get in the way they make it a lot harder to do something like that um thankfully we have yet another tool um turns out algorithms like merge sort um more generally what we'll call divide and conquer recurrences uh sorry algorithms there are lots of divide and conquer algorithms um very similar to merge merge sort that give runtime recurrences very similar to merge sort um divide and conquer means that usually whatever set of data you have you're dividing it into smaller groups there might be two there might be multiple um you're recursively calling your algorithm on each group potentially possibly your groups overlap so you need more recursive calls um and then after you get information about each of your subsets you somehow combine them back together to get your answer for the full list so merch sword is a good example there um you'll see some others in recitation and problem Set uh for other divide and conquer algorithms we're not so concerned with algorithm design in this class um but tools that will be used for Designing algorithms in the future uh are very appropriate uh and I'm I'm singling out divide and conquer algorithms because we have a really nice tool that helps us analyze the runtime of divide and conquer algorithms like this it's called The Master theorem um it's called The Master theorem and it looks like that uh it's a lot of text so I didn't want to write it by hand um what the heck is going on here well well the point is the following suppose we have a recurrence uh so we'll call it a divide and conquer recurrence is going to be one that looks like this it's going to be T of n equals some constant a Time T of the ceiling of n / B plus some function f of n where um a is let's see what are our conditions a is at least one and B is uh greater than one strictly greater than one and F of n is some function merge sort certainly fits this a is 2 B is 2 F of n is n minus one and the master theorem says if you have a divide and conquer recurrence that looks like this if you check just a a couple quick conditions on how quickly F grows I can use that to tell you a Theta bound on T it's not giving me an exact closed form it's giving me an ASM totic bound it's going to say that t of n um is in Theta of something there are three cases where the master theorem can tell you explicitly T of n is Theta of this nice simple thing all right so that's the idea let's not prove that right now but let's talk about what the three cases mean and where they come from uh and get some intuition for the master theorem okay and if we think about a divide and conquer algorithm like this it says well at the very top up um we have n elements in our list um and we're going to do some work with recursive calls that's the a * T of thing term but then we're going to do some work with these n elements themselves and that's F of n uh so we're going to do F of n work at this node and all the rest will be done in sort of the children if we're thinking about this as um the recursive call stack C tree um here's where we're dealing with t of n here's where we're dealing with t of n/ B and there are a different children in merge sort we're splitting into two halves we're making two recursive calls uh in this example maybe there are three recursive calls so a is three uh and at each of these nodes we are calling F of n / B right right and each of these is making some recursive calls each of these nodes uh is doing work F of n over b^2 then again F of n over B Cubed and so on all the way down F of n over B to the H however however tall this tree is um so each of these nodes is doing this amount of work in itself and the rest of the work is done recursively how many nodes do we have at each level well we have one node here a nodes here A Squared nodes here a cubed nodes here um a to the H nodes here at the bottom so we're getting more and more sub problems but the sizes of those sub problems are getting smaller and smaller um to put this in context for merge sort um at we called merge with n elements at the very top we had two cases where we called merge with n/ two elements four times where we called merge with n over four elements eight times we called merge with n over eight elements and so on and so on does this idea make sense this is sort of a um a conceptual view of all of the different recursive calls in our device divide and conquer algorithm and this sum here FN * 1 plus f n / B * a this time a squ this time a cubed all the way down to the bottom this is our runtime I'm being sketchy here I'm not dealing with the floors and ceilings this is this is just intuition anyway um so I'm I'm going to be a bit sloppy I apologize in advance um but this is conceptually our runtime first of all how tall is this tree what is this AG well we're going down until well with merge sort we're going all the way until we get a single element we sort a single element and that's our base case and we're certainly not going beyond that um well what does that say here that says n and B to the H are equal to each other in other words the height of the tree is log B of n all right and now we have the sum and and this is our runtime the master theorem says that this sum often has three familiar forms one of three familiar forms um in the first case in case one of the master theorem it says if FN is Big O of this thing ignore what this thing is if FN is Big O if FN is small so if F of n is small imag imagine in the extreme case FN is just one there's only a single step to do at each of these nodes well then our runtime looks like um 1 plus a + a 2 plus a cubed cubed plus a to the H if the fs are insignificant compared to the A's then our sum is basically this um and this is a geometric series which is basically as big as a to the H itself up to constant factors lower order terms we have a geometric series formula for that but basically it's saying if FN is small then this last term dominates the whole sum right we have these two conflicting effects where the fs are getting smaller because the inputs are getting smaller but the um the powers of a are getting bigger and so case one is describing the case where the A's grow faster than the fs shrink um and it turns out that in that case the runtime is basically just a to the H itself that t to the N is in Theta of a to the H which is a to the H is what a to the log B of n which by log rules is the same as n to the log B of A and that's exactly the result from case one if the A's overpower the fs then this last term wins and that basically gives us our ASM totic runtime um on the Other Extreme if F grows fast if f is Big so it grows really quickly um then the F terms over here are going to overpower the a terms um and we end up getting the opposite effect where this first term is the biggest and all the rest don't really contribute much ASM totically so if if f is Big then the first term in our sum dominates and we end up getting T of n um T of n is in Theta of F and that is what case three says so if F grows really fast then F alone is where most of the work happens in terms of our tree that means that the work we had to do in this top node FN that's the bulk of it and everything else we did down here is small in comparison all right now there's a case three I want to keep that tree so let's come over here sorry there's a case two the middle case of Master theorem that says what if these two effects kind of balance each other out um um and in this case um it's basically asking what if all of these terms so FN * 1 FN over B * a FN / b^ 2 * a^ 2 all of these terms in the sum what if they're all basically equal to each other so in the middle all terms of the sum are basically equal and so we end up getting that the sum is well there are H terms here H+ one terms and each of the terms is basically the same so I'm going to take this where it's a to the H times Well n over B to the H is one right so this is basically well H + one times uh what was that last term again a to the H * f of one if all the terms are basically equal then it's just the number of terms times this last one right um we're doing astics I don't care about that plus one uh F of one is a basic is a constant I don't care about that uh so this is this ends up being basically a to the H * H this is Theta of and if we remember what H was sorry if we remember what a to the H was that's n to the log B of A and if we remember what H was um times another Factor log B uh sorry yeah log B of n so a to the H is this power of n h is log of n it's basically saying in case two if things balance out nicely you just incur an extra factor of log n from what you would otherwise expect from case one and turns out merge sort is in that middle ground it's in case two um because FN here is n minus one for merge sort uh and it's asking is this Theta of n to the log base 2 of two that's the condition for case 2 to apply according to master theorem and yeah n minus one is in Theta of n this exponent is one so yes so we're in case 2 and this says that Master theorem of n is Theta of n to that exponent that exponent is one times an extra factor of log n so that's how we can use the master theorem just by pattern matching on the the shape ape of their recurrence a is 2 B is 2 FN is n minus one make sure it satisfies the condition for case 2 and in that case we can conclude uh that MN is Theta of n log n um which is the formula we found earlier um at least when we were only dealing with powers of two we had n log n plus a linear term minus a linear term and so this this matches what we had in general it's less informative it's only a Theta bound not an exact formula but it still gives us what we would expect which is nice um last thing I want to say about this merge sort does have gaps uh sorry not merge sort Master theorem does have gaps um if you have a divide and congr recurrence like this um you can't always just apply Master theorem sometimes um you have cases that don't fall into any of the three cases of Master theorem case one two and three are all false U those conditions are not valid so you can't apply Master theorem at all so Master theorem is not um snake oil that will solve every problem it'll solve many problems and in 006 in in 6121 um you'll be using it quite a bit and you'll see how how useful it can be in practice final thought um let's come back and compare some yeah I'll just use this compare some of our um recurrences and the growth rates that they describe so we had for example Fibonacci numbers uh FN is FN minus one plus FN minus 2 uh we had towers of Hanoi where hn was 2 h of n minus1 + one this was Fibonacci this was Hanoi we have merged sort um merge sort of n which is 2 m of n / 2 plus n minus one and other divide and conquer recurrences I'll use D for divide and conquer this is a * D of n / B plus F of N and if we take a look at what formulas we got for these this Fibonacci this was Theta of um like 1.6 to the N that 1 plusun 5 all divided by two to the N the Fibonacci numbers grow exponentially the towers of Hanoi grow exponentially merch sword is very small n log n um and the the main difference here is that in each of these recurrences we're making recursive calls to smaller terms but how quickly are those smaller terms decreasing in merge sort every time we make a recursive call we cut the size in half we Cut N in half so there are only logarithmically many times that we can cut it in half before we bought them out um so that's why our our call tree over there was only logarithmic height and so the runtime ends up being pretty small because there's not much of a tree to deal with by contrast with Hanoi and Fibonacci every time we make a recursive call we're only subtracting one or two and so we have to do that linearly many times like n times before we get down to the bottom and every time we make this call the tree is getting exponentially bigger it's doubling every time we do this basically uh and that's why in these two cases we're seeing exponential growth because the the tree is huge so just thinking about this recursive call tree which terms are calling which other terms and how far down does that go and how many nodes are there is a great way to get intuition um and a rough idea of how quickly these things are growing and a great reason why divide and conquer algorithms in general are often really good because they bottom out quickly that is everything I wanted to say thank you so much um we'll see you in recitation tomorrow and good luck studying for your quiz