Lecture 6: Asymptotics

Key Takeaways

good morning good afternoon welcome to lecture six um today we will be um finishing up our discussion of summations that binmore started for us last time and then moving on to uh asymptotic notation which is one of the bread and butter topics of algorithm analysis it's a really useful uh language and terminology that we can use to talk about algorithms and compare algorithms without getting bogged down in all the details but before we do that let's talk about sums and before we do that let's play with blocks um so here is a very cool question I have a stack of blocks you could imagine it's even taller than this I just grabbed six for now um and what I want to do is sort of lean them out over the table and well okay don't want them to fall that's that's Criterion one what I'm hoping for is to get this top block all the way off of this table um so none of the block is over the table so currently I'm not quite there we can see there's like an inch half an inch left um so seems possible maybe um so the rules are we can only have one block per layer we can't double up we can't do weird angles and whatnot it's just slide these left and right until you're happy and hopefully it doesn't fall um are we clear on the game nice who thinks it's possible raise your hand who thinks it's not possible all right well let's try hopefully I don't hurt myself um all right so I'm going to follow a nice greedy strategy so I'm going to start from the top and then I'm just going to keep sliding left as much as I can until it's about to fall over so if I go past the center of mass it's going to fall over so I'm going to go right up until that edge and stop and don't worry about the physics we're not here to talk about physics um but I'm going to do the same so let's freeze the top two layers and try again let's slide this over until it's just about to fall it feels right around there all right let's keep going all right yep okay it leaned a little bit I'm getting worried but we'll make it work let's see uh yeah okay and I think I might already be off the table which was my goal but let's let's go a little further why not uh yes okay and finally right around okay that's as far as I'm willing to go I think that is off the table it's possible yes thank you I cheated I put little marks on the back so I knew how far to go um but it is possible and now we can raise the question how far can we go can we get two lengths out here's what that would look like two lengths out can we get five lengths out 10 lengths out let's stop at 10 who thinks we can get 10 lengths out over the table who thinks 10 is possible who thinks 10 is not possible all right you're both right um I'll explain why in a second uh let's not have an accident during lecture there we go so turns out I'm not going to prove this now but turns out that greedy strategy I just used is the best you can do uh it's not too hard to prove it's in the textbook if you're curious um but let me tell you what that greedy strategy ends up looking like um if I have some chalk here we go so as you might imagine this first block sticks out half a unit over the next block so its Center of mass is right here on the edge um by the way I'm I'm imagining these blocks are one unit wide um the next one right here that's a quarter unit because that's what it takes to put the center of mass of these two blocks combined right over the edge of the third block turns out this right here is a sixth this right here is an eighth and all the way down until the very n until the the it is the the very n the nth block has a gap of 1 over 2 N so this is block 1 2 3 four five all the way to n turns out this is the best you can do which means the distance you can get with n blocks is 12 plus a 4 plus a 6 up to 1 over 2 N which let's factor out the 1/2 just because that's cleaner that's the sum K = 1 to n of 1 over K this is also known as 1 12 time the N harmonic number harmonic number this is a standard name for this sum 1 plus a half plus a thir plus a 4th up to 1/ n um and half of that is exactly as far as we can get with n blocks right so now the question is how how big does this get if I have a thousand blocks how far can I get if I have a million blocks if I have an unlimited number of blocks how far can I get which means we're asking about this sum this harmonic number I want to know about that sum do we have a closed form for it no not really but as we saw last time we don't especially need a closed form if all we care about is approximation so let's approximate that sum so we have our sum hn which is defined as the sum of the first n reciprocals and we want to approximate it and can someone remind me what tool we have to approximate sums like this yes the integral method absolutely so this function 1/x is a decreasing function and so we have our integral method our integral bound that says that sum hn is between the integral plus the last term in this case the last term is sorry plus the first term one and the integral 1 to n of 1x DX plus the last term which is 1 n so this bound is integral plus F of n less equal to the sum is less equal to the integral plus F of one and I claim this bound is really easy to remember um like I plus FN I plus F1 are pretty simple things and also if you if if you're dealing with an increasing function instead of a decreasing function it's the same two bounds but they're on the other side just because if it's increasing then FN is going to be bigger than F1 so certainly FN should be on the bigger side but this is a really clean bound um and and we can write out what this what this says integral from one of 1 /x DX from 1 to n that's Ln of n let's assume we can do that um without writing it down I you don't want to see me integrate live that will go poorly um but what this says is that um Ln of n + 1 n is less equal to hn is less equal to Ln of n + one all right which is a pretty nice bound um but we're really interested in well first let's answer our question um if we want to get 10 blocks out then we're asking for hn to be about 20 which means we're asking for these bounds these bounds are kind of basically Ln of n right the one over n and and the one don't affect things all that much so really hn is basically Ln of n yeah this is I'm being imprecise here but we'll be more precise about it in a bit but for now it's about Ln of n which means to get 10 blocks out we need hn equals 20 because the distance was half hn uh which means we need n is approximately e to the 20 we need that many blocks uh we need about four 400 billion blocks to get 10 blocks away from the table um each one is about an inch and a half thick so that gets us to carry the one 40 times the distance to the Moon um probably going to run into some other issues before you're able to stack something that tall um so mathematically yes we can get 10 blocks over the table physically not so much that's why you're both right um but all of this was really an excuse to get us here hn is approximately Ln of n plus some smaller terms um and let's try to make that a little more precise what do we mean by that what do we mean by hn is approximately Ln of n well let me give us a definition [Applause] let's write that F of n that's a Tilda F of n is Tia of G of n so this is asymptotically equivalent so f and g are ASM totically equivalent if and only if the limit of FN over G of n as n goes to Infinity is one if you divide them and get a limit of one then these functions are ASM totically equivalent and in this case well we have these nice bounds on hn well let's take that inequality let's divide all three sides U by Ln of n what are we going to get we're going to get 1 + 1 over n Ln of n is less equal to hn over Ln of n is less equal to 1 + 1 / Ln of N and as we take the limit as n goes to Infinity the left side looks like one this thing in the denominator gets big so one over that gets small the left side goes to one limb n goes to Infinity of hn over Ln of N and on the right side we have one plus something that goes to zero cuz that denominator gets arbitrarily big so the limit equals one I learned this in high school as the squeeze theorem if you can bound your limit between two things that are the same then the limit is that thing um but what we've concluded here is that hn ided Ln of n limits to one as n goes to infinity and therefore uh sorry hn over Ln of n equals 1 which implies that hn is ASM totically equivalent to Ln of n so when I wrote approximately Ln of n up there probably what I meant was this much more precise thing that these are approximately equal in the sense that their ratio limits to one does that make sense nice all right um now before moving on to ASM totic fully uh there's one other topic in summations that I want to talk about which is not summations at all but products um and really what I want to say about products is that often you can analyze products as if they were sums turn into sums for example if we care about n factorial which is uh here 1 * 2 * 3 all the way up to n who can tell me how to turn this into a sum it's something written on this board yes yeah take logs perfect what happens if we take the log of both sides so log of n factorial uh here let's let's be precise we'll call it Ln of n factorial the natural log so log base e well remember when you take logs products turn to sums in this sense this becomes Ln of one plus Ln of 2 plus Ln of 3 plus Ln of n so if we want to understand n factorial we could just as well understand its log and its log is the sum of a bunch of other logs and now we have a sum and we can use all of our summation uh tools on this sum to get information about n factorial for example we could use the integral bound Ln of X is an increasing function this time so we use the increasing version of the bound uh which says that uh here we go so Ln of one plus the integral from 1 to n of Ln of X DX is less equal to the sum of Ln of K K = 1 to n Which is less equal to Ln of n plus the same integral and if we do out that integral like I said you don't want to see me integrate live I will get it wrong um but if we trust my notes this integral comes out to um n Ln of n um minus n + 1 Ln of one is zero thankfully keeps things simple and so what we end up finding is that n Ln of nus n + 1 is less equal to this sum which as we know is Ln of n factorial Which is less equal to this same integral plus another Ln of n n + one Ln of nus n + 1 so we were able to translate this product into a sum and then use our summation tool um to get get us a bound on Ln of N and if we want to bound on N factor sorry Ln of n factorial I misspoke if we want to bound on Ln I misspoke again apologies if we want to bound on N factorial itself we can just undo the log by exponentiating all three of these take e to this e to this e to this that preserves our inequalities and what we'll end up getting after we simplify it a bit is that n the N over e to the N minus1 is less equ to n factorial is less equal to n n +1 over e nus1 so same denominator on both sides we have n the N versus n the N plus one on the two sides so the two bounds are off by a factor of n not as good as we had last time we are off by an additive one less than that even but a factor of n isn't bad for a number as big as n factorial this isn't a bad bound um is everyone clear on how we got to this bound wonderful it's not the best bound we have ow um let me tell you uh two better ones oh sorry before that is this bound enough to say that n factorial is ASM totically equivalent to one side or the other I see some head shakes can someone tell me why that's all right I'm tired too um these two bounds are too far apart from each other they're a factor of n apart and so we can't say that this divided by this limits to one or this divided by this limits to one because we have an N range in there to account for and N grows to Infinity so we don't know where in this range and factorial lies and so we can't pin down that limit going to a specific value so this isn't tight enough to say that n factorial is ASM totically equivalent to something um but that's where Sterling's approximation comes in um Sterling's approximation says that n factorial is ASM totically equivalent to theun of 2 piun n * n e to the end we're not going to prove that but it's really useful you should write it down you might end up using it in the problem set hint hint um extremely useful um but remember this is this is just saying that n factorial divided by this limits to one so when n is really really big the quotient of these two things is close to one what it doesn't tell us is what happens for smaller values of n what if n is just 10 can we use this to approximate 10 factorial how about 100 how about a thousand I don't know this doesn't really tell us it doesn't tell us how good an approximation this is all it says is that it's good in the limit um so let me tell you an even stronger bound uh I don't know who to credit this bound to but it's even more precise than this one uh it says that n factorial divided by this Thing 2 pi n n / e to the N so this is a thing that Sterling tells us limits to one and now we have some bounds on how quickly it does that e to 1 over 12n e to 1 over 12n + 1 so not only does this limit to one it does so very very fast exponentially fast and this we can plug in specific values of N and see okay how close is this approximation to n factorial this is an effective bound one we can get actual approximations out of this is just a limiting bound but of course this is a lot more complicated sometimes we don't want to deal with all these details and this is enough sometimes even that's too much and that's enough so pick your level of granularity as you need it we're also not going to prove this bound um that's even harder than Sterling um but it's true and useful questions on that all right so that's everything I wanted to say about summations and now let's jump back to astics we already started talking about astics with um ASM totic equivalents um and let's do an experiment uh let's do an exercise um we've already seen a sorting algorithm we saw swap sort um in lecture four where you you have a you want to put it in sorted order and you do that by finding a pair that's out of order and next to each other and changing their order and you keep doing that keep finding pairs that are next to each other and out of order until the whole thing is sorted uh and we saw so I'm going to call that swap sort we saw that the the number of swaps that you'll need to make is exactly equal to number of swaps is exactly equal to the number of inverted pairs so the number of pairs of two numbers in our list where the big one is somewhere to the left of the small one so they're in the wrong order and we'll eventually need to change order um and well how many inverted pairs could there be in the worst case well in the worst case every pair is inverted one the number one is is inverted with 2 through n and two is inverted well it's already inverted through one we already counted that but two would be inverted with three through n those are new in inversions that we haven't counted yet uh three is inverted with everything after it plus one so this is the worst case number of swaps we'll have to make um and as we as we've seen we know how to compute this sum this is n minus one * n / two also known as n ^2 over 2us n /2 and this I'm going to call S of n we have our function s of n s being reminiscent of swap sort it's a function describing the worst case number of swaps we'll need to make in swap sort on Thursday we're going to see another sorting algorithm called merge sort uh I'll describe it then I'm not not going to describe it now that's we don't need to know what it does or how uh we just need to know how fast it runs how many comparisons does it have to make um so let's define M of n this is going to be n log 2 of n minus n + one uh and on Thursday we'll see that uh that merge sort takes at most this many steps to complete and now I want to know between these two function between these two algorithms which one is a better choice for sorting lots of numbers if these are the two algorithms I have to pick from and I know their run times they're n^2 / 2 - n / 2 versus n log 2 N - n + 1 this is s this is M which one's the better choice um and sorry that was rhetorical um let's talk through it um let's do sort of an eye tracker to see which parts of the various formulas we look at when making this comparison uh and just to emphasize I'm I'm most worried about what happens with lots and lots and lots of numbers I'm trying to sort a million a billion a trillion numbers um so I'm interested in what happens in the limit as n gets really really big if we're trying to sort 10 numbers well I don't care that's what however you want to do it is fine 10 is small um but with big data no one uses that term anymore I don't know why um with lots and lots of data these these differences matter so let's see how we make this comparison what are we going to look at well the first thing my eye goes to is this right here which is to say that's the last place my eye goes I don't care about that term the n^2 over two is much more important um by the time we're up into the N squar it's an a little measly n / two doesn't really make much of a difference right intuitively anyway I haven't made this precise but the first thing that I do when looking at this is I just completely ignore that N /2 I don't care likewise I don't care about this n minus one that we're subtracting the N log n grows faster than that uh so with big n this is going to dominate and this is just going to be a rounding eror so that's the first thing I do ignore the lower terms now I'm just comparing n^2 2 versus n log n the next thing I ignore is this over two I don't really care I don't care that we're dividing by two if this had been n^2 divided a th if this had been a million time n log n is that really going to change my determination for small n values yeah those constants matter those lower terms matter for that matter when n is small but in the limit as n gets bigger and bigger that that extra factor of n is going to overtake that extra factor of log n no matter what constants we throw in there right so I don't really care about those as well really all I'm focusing on is the N2 versus the N log n does that make sense um every everything I just did was completely informal I haven't Justified any of that but we will um but first we need to motivate it and there's one other reason uh one other motivation for not really caring about this constant Factor uh with swap sort we were counting what were we counting the number of actual swaps we make um we didn't say how you're finding those swaps but there are there are clean ways to do it so that the number of steps you take um in the whole algorithm is about n^2 over two um but wait we have to do more steps than just the swaps right we have to like read numbers from memory we have to um compare them we have to write numbers to memory we have to increment our index we have lots and lots of instructions that we have to do during this algorithm so maybe it's just those four instructions every time we make a swap so it's really four times n^2 over two instructions right instead of counting swaps we're now counting instructions and I'm being very loose about that but doesn't matter um okay well not all instructions are are created equal um every instruction on your CPU happens on a a fixed regular clock um and maybe ads take when we have to increment our index maybe that takes I don't know five Cycles uh and comparisons take 15 cycles and read and write to memory take I don't know 200 Cycles because memory is a lot slower than arithmetic I made up these numbers by the way these aren't real don't trust me on that um but in this fiction if we're counting Cycles instead of instructions we're doing like one of each of these per per round or something um so really we're doing if I did my math right about 210 * n^2 Cycles okay so we're changing our unit we're measuring different things um let's measure something else say we're on a 2.4 gigahertz processor 2.4 gigahertz processor how long does this take in in real life time well it takes about 210 divided 2.4 billion seconds uh sorry times n^ 2 seconds right so if we change what we're measuring if we change our units instructions Cycles seconds the thing that stays is the N squared the constant changes but it's always proportional to n^2 get a better um CPU um higher throughput your time decreases but only by a constant factor it's still proportional to n^2 this is another reason that it's useful to be able to just ignore that constant I don't care what it is because I don't really care specifically what units we're measuring um I just care that it's proportional to the right answer does that kind of make sense why we might want to throw away information like this because it's easier to think about n squ is easier to think about than all this other junk cluttering it up um one more one more motivating example what's the population of the US does anyone know just off the top yes about 330 million absolutely about 335 um what is it um I wrote it down 893,00 is that more useful than what your colleague said 330 million I don't get any more information out of that in in my measly little brain 330 is I already have trouble conceiving of a number that large I don't need all the extra detail right same for algorithm analysis I want to be able to just focus in on the parts that matter and have the option to strip off the parts that don't so that's our motivation we want to be able to to communicate less information about the run times of our algorithms or about other functions while still being mathematically precise about it and making sure we know what what was lost what information are we retaining and what information are we losing never to to get again because sometimes you need that extra information throwing away all this stuff is a great like broad Strokes way to compare um certain algorithms but if we have two algorithms that are both proportional to n log n oh well now that constant Factor really matters and we better go and figure out what it is likewise if we're comparing the US population between this year and last year well last year it was also about 330 million in change so if all we know is 330 330 we can't compare them we don't know if it went up or down or by how much um turns out last year it was about 334 um2 million um million so now we had a reason that we wanted more information wanted more precise measurements of the population than just 330 million um but wait until you have that reason let's let's do the broad stroke stuff first and often that'll be enough does that make sense as a motivation for why we want to throw away information about our functions nice so let's talk about how to do that uh and the the main one we're going to talk about is called Big O notation the idea this is not yet a definition still an idea uh we're going to say that F of n belongs to Big O of G of n weird notation but you'll get used to it um when well roughly F of n is less equal to G of n roughly if we say f is Big O of G what we're saying is that an algorithm with runtime f ofn is at least as fast maybe maybe much faster than an algorithm with runtime G of n with some caveats with caveats first of all let's throw away constant factors if we have to scale G up by 10 and now G is bigger that's fine that's still Big O uh and likewise we're going to ignore I don't care what happens for small numbers ignoring small n don't need them I care about what happens in the limit what are the trends when n gets bigger and bigger so these are my intuitions um also ignoring lower order terms I still haven't defined that either by the way but we talked about them over here like the N /2 was a lower order term compared to n^2 over two I want to be able to ignore things like that so this is still not precise let's finally finally make it precise after this question yes oh no I want to give the definition thank you uh okay so our definition our actual final definition of Big O is the following um f of n is in Big O of G of N means okay there exists a number greater than zero and there exists a natural number and not such that for all n greater or equal to n f of n is less equal to C * G of n um and if we're not being if we're not restricting to positive functions sometimes we put an absolute value in there that's less important the other parts of the formula are the more important parts there exists a constant factor C so that f is less equal to C * G that's this ignoring constant factors um there exists an N so that all n greater than n greater than or equal to n so there's some starting point so that after that starting point this this inequality is always true and I don't care what happens before n knot um so that's this point here ignoring small n small n are the ones smaller than n then n not um n Subzero is sometimes pronounced n not n a u g HT uh question oh gosh yes thank you uh G of n not the line I wanted to mess up today thank you so much yes F of n is in bigo of G of N means roughly that g is bigger if you're allowed to scale it by a constant and throw away small values all right yes yeah um so um FN is in bigo of g ofn means that there's some positive value C and some starting point n KN so that for all input values that are bigger than n not or equal FN is less or equal to C * G of n uh so this is extending our intuition that FN is small that f is smaller than G but you're allowed to put in a constant Factor if you need to before that's true yes um yeah so why are the question was why are we putting in a constant Factor when the whole point was to ignore the constant Factor we ignore it by saying that there exists a constant factor and I don't care what it is as long as some constant exists that makes this formula true then f is Big O of G um so we put it in so that we can not care what its value is um yes can the Conant between Z and one can the constant be between zero and one good question sure why not um if you want to make G smaller and it's still bigger than F go for it no problems there all right let's do some examples hopefully some concrete examples will make this a bit clearer um let's say F of n is n g of n is n 2 is f of n in Big O of G of n so is n in Big O of n squ well if this is supposed to say that the thing on the right is bigger then hopefully we would want n squ to be bigger than n and thankfully it is um in this case we don't need the C to help us f is already smaller than g n is already smaller than n^2 n is less equal to 1 * n^ 2 for all n for all natural numbers n so this would be our n value of zero and this would be our C value of one all we have to do is show that some not exists and some C exists to make this statement true and then we're able to conclude that n is Big O of n^ s question um ah good question um don't we need n not to be one because this inequality is false when n is between zero and one um excellent point um I'm going to give two fixes for that first of all we could change this to one absolutely that's fine um the other change is that let's go ahead and assume that f and g are mapping the natural numbers to the real numbers so the inputs are just 0 1 2 3 and not all the real numbers in between often that's what we use when we're analyzing algorithms because we don't have fractional inputs to to algorithms um you could absolutely do all of this analysis and all these definitions if your domain was the reals instead um but let's let's deal with Naturals great uh great point also in fact why don't we go ahead and assume that all our functions are positive valued um then we won't need that weird absolute value you can still do a lot of this um if you allow some of your functions to be zero or negative but but let's not clutter our minds with that today let's go ahead and assume that all of our functions map the natural numbers to positive real values yes um the question was is this statement equivalent to saying that f is ASM totically equivalent to some constant multiple of G that's a great question the answer is no uh and I'll challenge you to think about why um let's do a few more examples uh actually I want to come back over here all right F of n is n s g of n is n F of n in Big O of G of n is n s smaller than n if you're allowed to scale by a constant and this time the answer is no um an algorithm that takes n s time certainly wants to look slower take longer than an algorithm that takes n time so if Big O is going to be useful for algorithm comparison we hope that this would be f and thankfully it is and let's talk about why um so if n s were in Big O of n we would need there exists a c and there exists an N so that for all n greater equal to n KN uh let's see n^2 is less equal to C * n right I just wrote out the definition so we would need to be able to pick some constant and some starting point so that after that starting point n^2 is always less equal to that fixed constant times n but no matter what constant we pick when n gets big enough this inequality is always false we can't pick a constant big enough so that this remains true for all n for all big n if we pick C is a th then from a th and one on this inequality is going to be be false if C is a million then from a million and one on this inequality is false no matter what C we pick the left side will overtake the right side right and increasing n not doesn't help with that even if we ignore more small values all the big values fail you can't ignore all the values so um can't choose c big enough all right so thankfully n^2 is not Big O of n uh let's do another example uh is n^2 + 3 n + 7 is that Big O of n^2 we said we wanted to be able to ignore lower order terms ignore things that look smaller um well this one ends up being true and let's see why well we can write right there are lots of ways to prove this um I'm going to be extremely wasteful and do the following n^2 + 3 n + 7 is less equal to n^2 + 3 n^ 2 + 7 n^ 2 all I'm doing is saying this n is less equal to n s this seven uh this seven is less equal to 7 n 2 right and this equals 11 N2 and so we've we were able to show that this function is less equal to 11 * this function for all n so we can take n equals z and C is 11 and we're done so lower order terms like this at most Force us to pick a bigger C if we want but again we don't care about the specific value of C as long as such a value exists so we can always swallow up these these smaller terms just absorb them into that c constant yes good question what if we didn't know whether one function is bigger than another function just by looking at it um well that is that is our task um we're trying to compare these functions and if we don't know which one's bigger then we better figure figure that out before we get our answer so I don't have a general way to do that um I'm I can just say that it's necessary um let's do another example 3 n is that Big O of n minus 4 we were also saying we'd like to be able to ignore lower order terms on the right hand side and intuitively that n minus 4 that minus 4 term doesn't really change anything this should still look like Big O of n right and we can make that precise um nus 4 is greater than or equal to n minus well n /2 when n is at least eight all we're saying is that the N function overtakes the four function excuse me so this four we can pretend it's a n/2 we've lost a lot but it doesn't matter because this is still a constant time n so this function is bigger than a constant time n and we can use that to do our proof um 3n let's see 3 n is less equal to 6 * 12 n Which is less equal to 6 * um that's that so nus 4 when n is at least eight and so now we have our C and we have our n 3n is less equal to a constant times n minus 4 which is the function we care about so in the same way we can ignore lower order terms on the right again by just swallowing them into the constant C at the cost of a slightly higher constant C does that make sense nice now some of you might recognize the definition of Big O is looking a lot like Epsilon Delta definitions of limits and you if you haven't seen those that's fine don't worry it doesn't matter but um they are similar Concepts and we have a useful fact that um compares the two that I will write down now theorem so we're assuming that f and g are functions from n to r+ to the positive reals if the limit of f of n over G of n as n goes to Infinity exists and equals infinity so if F over G gets arbitrarily big then there's no constant you can choose to bring G back up to F because F gets um bigger and bigger compared to G then f is not in Big O of G likewise if the limit exists and is less than infinity then f is in Big O of G and finally if the limit does not exist inconclusive F might be Big O of g f might not be Big O of G if the limit doesn't exist the limit doesn't help us get an answer but if the limit does exist it tells us the answer if the limit exists it tells us yes in this case no in that case so this is a one-sided test if the limit exists you're golden if the limit doesn't exist you have to fall back to the more complicated definition all right uh I'm not going to prove that it's uh not difficult to prove it's in the notes if you want to read about it but let's use it uh let's see is n^2 in Big O of a million n well how do we test that let let's use this limit test we can look at the limit of f over G of n^2 over a million n as n goes to Infinity n^2 over a constant time n that's infinity and so the limit exists it equals infinity and that tells us that f is not Big O of so very often if you're working with actual functions and you can actually take the limit and the limit actually exists you get your answer much faster than if you were trying to think through the C's and the nend knots and whatnot um another example is a thousand in Big O of n^2 what the heck do I even mean by that a thousand isn't a function of N I thought we were talking about functions of n right well it is a function of n if you want to think of it that way let me plot it for you let let me plot the N SAR this is uh n s and let me plot the function that takes every n and sends it to the value a th right right so this is f of n which is defined as no matter what your input is ignore it and spit out a thousand it's the constant function that spits out a thousand so when you see something like this when you see a constant in one of these ASM totic uh notations it's really talking about the constant function that sends every n to that value and in that sense now we can ask is this constant function Big O of n^2 and we can answer it the same way as before by looking at the limmit of f over G the limit of f which is a th over G which is N2 2 as n goes to Infinity that limit is zero so yes yes a th is Big O of n^2 when the limit exists it's great we're very happy more examples we're trying to be permissive with constant factors right so is it true that 3 to the N is Big O of 2 to the N if constant factors don't matter then maybe I maybe I should be able to change this constant to a different constant and it won't really change according to Big O is this true and this time it's not because we can try our limit test the limit of f over G that's 3 the N over 2 the N that's the limit of 1.5 to the N which is infinity so the answer is no changing this two to a three isn't changing by a constant factor it's changing by a constant Factor n times and that n um gets bigger and bigger so the difference between the two sides gets bigger and bigger so this is uh this is a no all right um some quick notation I've been writing F of n is in Big O of G of n um and this is my preferred notation this is set notation f ofn is an element of this set so big O of G of n is really a set of all the functions that have the relationship we described this I think is the clearest notation it's what I it's what I encourage you to use but you're going to see some other not ations for example you're going to see F of n equals Big O of G of n which is just super confusing I don't like it I don't recommend using it but it is pretty standard in the math community so you are going to see it so I have to warn you um you're also going to see f ofn is less equal to Big O of G of n this is again an an abuse of notation FN is a function this thing on the right is a set but it's communicating the same idea in fact it's emphasizing that same idea Big O means that g is bigger than F roughly and so this less equal sign is just a reminder of that so this is also good notation I recommend it if you want to use it um you're also going to see f ofn is Big O G of n instead of is an element of Big O of G of n that's also fine um pretty clear unambiguous all of these mean exactly the same thing sound good all right what you should not write ever um so we were saying uh for example we know that merge merge sort MN is less equal to Big O of uh swap swort of n those two functions we looked at earlier the n s one and the N log N1 so this is a true statement right the one was smaller if we want to say that the S is bigger should we say that SN is greater or equal to Big O of M of N and from the way I'm phrasing the question the answer is probably no um no we should not write this that's like saying you must be at least 60 in or less to ride this ride how tall do I need to be 6 it's not correct um Big O is functions that are smaller than M of N and if we're supposed to be bigger than some functions that are smaller than M of n it doesn't make sense don't write that however this is still an idea that we often want to communicate s is bigger than M we want to emphasize that s is bigger than M instead of emphasizing that m is smaller than S and we have different notation for that instead of a big O we use S is big Omega of M of M of n so big Omega is just the the reverse of Big O U we have our definition f is in big Omega of g means that g is in Big O of f let's take that as our definition and we're getting lots of symbols around here so let's see if we can compile them into a table so we started with f is ASM totically equivalent to G um the definition is that the limit as n goes to Infinity of f of n over G of n equals 1 the idea the intuition is that f is equivalent to G um up to lower order terms lower order terms that was our intuition then we had Big O F is in Big O of G the definition this time was there exists C greater Z there exists uh n KN greater equal zero so that for all n greater equal to n uh F of n is less equal to C * G of n so that's our definition our intuition is that f is less equal to G up to constant values up to constant factors and I'll say lower order terms and ignoring small numbers but that one does that too please don't confuse the intuition with the definition and please don't use the intuition language in your proofs use the the definition language in your proofs so think about it this way but write it this way and we Justus defined f is in big Omega of g means the definition is that g uh is in Big O of F and the idea is that uh G is greater sorry is that f is greater equal to G up two constants and lower order terms right so we have a a more permissive version of less equal that's Big O A more permissive version of greater equal that's big Omega questions about that all right um what did I want to say about that let me Define one more while I'm over here f is in Theta of G that's a capital Greek Theta um and this is going to mean roughly uh f equals g sorry f is approximately equal to G up to constants and lower order terms and the definition is just f is in Big O of G and G is in in Big O of f so f and g are roughly within constant factors of each other in both directions uh and I want to call attention to this first one versus this last one ASM totic equivalence said that f and g were roughly equal up to lower order terms Theta says that f and g are approximately equal up to con constant factors and lower order terms Theta is more permissive than Tilda um where did I leave off on the boards over here yeah over [Applause] here B so we can say that our our s of n function our our swap sort our simple sort function n^2 over two and junk uh is in Theta of n^2 it's both greater than um n^2 and less than n^2 if you're allowed to put constants in there all right um what's not true is SN is ASM totically equivalent to N2 this is false um because if you if you take this and divide by that and take the limit you get two not one that n^2 over two matters um so this cares what that constant factor is this doesn't um this would be a true statement if you put the one/ half in there so the ASM totic equivalence does single out that constant Factor whereas Theta ignores that constant Factor was there a question um okay so the question was um similar to your previous question um does f in Theta of g mean that there exists a constant such that f is Tilda of C time G that's a really good question um and the answer I will tell you um is actually the other way if there's a constant factor between them and then the limit goes to one uh then absolutely they are Theta of each other but there doesn't have to be that constant factor and let's look at an example of that let's see 3 + sin of n is that Theta of one 3 + S of n we want to know is it Theta of one so is it bounded below and bounded above by some constant times one and in this case yes it is because 2 is less equal to 3 + s of n Which is less equal to four so there's we can scale this by a constant and it's always smaller than our function we can scale this by a different constant and it's always bigger than our function and that's the condition we needed for Big O on both sides so yes this is true however you're never going to find a constant where 3 + sin of n is ASM totically equivalent to that constant * 1 because that ratio isn't going to have a limit this is an example where the limit of 3 + sin of n over one does not exist the function just keeps doing this forever and never settles down and picks a single value so this is an example where the limit between them does not exist but still they are Big O of each other right um in fact in both directions we mentioned earlier that if the limit doesn't exist sometimes um they are not Big O of each other can anyone think of an example of that two functions uh whose ratio does not have a limit where f is not Big O of G little tricky uh let me give Let me Give an example uh let's say that F of n is n and G of n is one when n is odd and n^ S when n is even so if we plot F of n is just n g of n um is following one or n s depending where you are and it's sort of zigzagging between the two if you scale G by a constant you'll never be able to make it bigger or smaller than n um because the the ratio is is too much depending whether you're looking at the odds or the evens um you can't scale it in either direction so F in Big O of G is false and G in Big O of f is also false um and it was specially constructed so that limit of f over G as n goes to Infinity um does not exist because on the odds F overg grows really big and on the evens f over G is really small sound good nice all right speaking of getting really big and really small um there are two more ASM totic symbols that we need to add to our table uh and those are the last two I promise um the first one I want to Define is called little o so f is in little o of G the definition is going to be that the limit as n goes to Infinity of f of n over G of n equals z and the intuition is that f is much much much smaller than G if we want to say that f is insignificant compared to G that's just saying that the limit between them goes to zero right so that's our that's going to be our definition of little O notation wise little o and Big O look similar um so when you're writing those by hand please try to exaggerate uh I I like to write My Little O's really really small um like even smaller than other lowercase letters and I like to write my Big O really really big not like hilariously big but noticeably big so no one's going to confuse it for a little o um annoying notation that those look similar so just be aware and uh try to correct for it um and there's the reverse f is in little Omega of G that's the lowercase Omega compared to the capital Omega up there means the opposite that the limit as n goes to Infinity of f of n over G of n equals infinity AKA G is in little o of f so if we want to stress that f is much much bigger than G we can use the little Omega notation that f is much much bigger than G all right and these are useful I mean for the same reasons the other things are useful they're giving us more vocabulary to compare functions um in ways that let us think about only a small amount of information about each function um but little o is sort of exactly what we meant by lower order term uh claim uh that f is going to be ASM totically equivalent to f plus h if and only if H is in little o of f if and only if the limit of H over f is zero h of n over F of n as n goes to Infinity so lower order term just means the ratio goes to zero in the limit uh and that's consistent with what we've been saying with lower order terms all throughout today lower order term just means limit goes to zero and we have a special notation for that o um f is ASM totically equivalent to f plus h means that according to ASM totic equivalence the H didn't change anything the H was insignificant so that's another way to think about what we mean by lower order term make sense nice um cool last thing I want to do today is a big old warning Asim totic notation and induction don't mix easily if you're ever doing astics and induction in the same same breadth step back take care let's see what can go wrong all right so theorem question mark 2 to the N is Big O of one now we really really hope this isn't true in algorithm terms this is an algorithm that takes exponential time this is an algorithm that takes constant time certainly this one's supposed to be the smaller one so 2 to the N should absolutely not be in Big O of one but let's prove it anyway proof by induction let's do our base case n equals z so 2 to the 0o is in Big O of one that's just that's saying that one is Big O of one which it certainly is all right inductive step uh assume n is at least zero and that 2 to the N is in Big O of one want to show 2 the n + 1 is in Big O of one but but well 2 the n + 1 that's 2 the n + 2 the N which is Big O of one plus Big O of one by our inductive assumption and that is Big O of one so 2 the N plus one is Big O of one and we're done we proved our induction what the heck went wrong yes 2 the N plus 1 is not equal to 2 the N plus 2 the N um it is 2 the N plus 1 is just 2 * 2 the N um which I can write as two copies of two to the UN added together um I will tell you the mistake isn't in this line at all I know I did something really sketchy right here what do I mean by o of one plus o of one that's a set and that's a set why am I able to add them also I accidentally Ed that when I meant set membership instead of equality but actually this is kind of fine this is an abusive notation we didn't really Define what this means but all we mean by it um o of f plus o of G this is just the set of functions um a plus b where a is in o of F and B is in O of G this just means the set of things you can form by taking a one function from this side and one function from that side and adding them together um and it is true that in this case o of one plus o of one equals o of one as sets so that's fine that's not the mistake yes all right um so let me reiterate um I proved that every number two to the N is Big O of one itself but I didn't prove the function 2 to the N is Big O of one that's exactly the idea so let's see what we actually proved here this here is a correct proof by induction but what is it proving it it's a proof of what we what we really proved is that two to the is is that one is in Big O of one that was our base case we proved that two is in Big O of one four is in Big O of one eight is in Big O of one 16 is in Big O of one we proved that each individual power of two when considered as a constant function is Big O of one so every constant function is Big O of one yeah that's true we proved it by induction that has absolutely nothing to do with the two to the N function has absolutely nothing to do with the definition of 2 to the N being in Big O of one so we didn't prove the right theorem we proved something completely irrelevant does that make sense so this is why induction and astics don't mix when you see a function of n you can't break that function into individual constant functions one for each n value you can't induct on N you have to treat that function holistically as a single function sound good all right um does anyone have any questions about anything we talked about yes excellent question does little o imply Big O uh so if we know that f o sorry if we know that f is in little o of G does that imply that f is in Big O of G yeah yeah absolutely um and let's let's think about why this so little o is defined by uh let's see f um sorry F over G that limit equals zero but remember we already said early on that if the limit exists then it tells you whether or not it's Big O and in this case the limit is zero so it tells us yes we are Big O yep so little o is absolutely stronger than Big O it implies Big O um we we certainly don't go the other way um likewise um uh let's see there are lots of other implications between this between these so um Theta if f is a Theta of G well that certainly implies Big O and big Omega bec