[MINI] Max-pooling

Key Takeaways

[Music] Data Skeptic is the official podcast of dataskeepic.com bringing you stories, interviews, and many episodes on topics in data science, machine learning, statistics, and artificial intelligence. [Music] So, Linda, today's episode is going to be called Max Pooling, but it's really going to be about all of pooling. We're going to talk about pooling in general. Great. I was just going to ask, what is pooling? Well, you know, it's that thing you do when you look on satellite maps to find out what buildings have pools and then you sneak in illegally and you swim there. That's not illegal. That is a thing she does, though. It's just called trespassing. Tell me, before we get started, tell me about that idea. I thought that was hilarious when you told me this. So, we live in Los Angeles. It gets hot here. So, sometimes it gets hot and we don't have a pool. I used to live in Palms and I didn't want to drive, so we knew there were apartment complexes nearby. So, we just went on Google satellite and zoomed in and figured out which apartment buildings had pools and then waited near the doors. Yet another example of Google Maps being used by criminals. They should just take that service down. And then we would get in and we would had the pool all to ourselves cuz all of those apartment pools are underutilized. Well, that's not exactly the pooling we're talking about here today. We're talking about bringing many things together, which I guess is like a different definition entirely. In the case of neural networks, what pooling means is you take a bunch of neurons and you aggregate them in some way. Now, as I said, max pooling is the most common one. That's where you basically just say who has the maximum value of all the inputs and you say that that is now the new input for the group. Who has the max input output actually? Output. Yeah. And that is the what? That is the value you assign to the group. Why would you do it to the max instead of the medium? Ah, well that's a good question. Okay, so I thought of a nice analogy. What if you um put out like a a broadcast on Facebook to all your friends? Let's say you saw everybody at some picnic earlier in the afternoon and you're like, "Oh my god, friends, I lost my keys. Has anyone seen my keys?" Now, if everyone replies back, "No, no, no, no, no." and only one person says yes and it's like well the average person didn't know where your keys were that's not very helpful right in this case if any single person says yes I found your keys that's really good so what you're kind of looking for is even if just one element one of your sensors in this case your sensors are your friends in a neural network it's your set of inputs which are the outputs of some other layer if any one of them has a high value like yes yes I know where it is you want to assume that the group as a whole has some benefit benefit or has detected a feature. M okay. But now on the other hand, you're right. The average might be better sometimes. What if you were going to go on Facebook and ask your friends like, "Hey, do do these outlandish boots that I bought uh are these okay to wear to a fancy cocktail party?" You know, if one person says yes, probably that you don't want to take it on their authority that you should wear them. You want to see what the average of the group thinks then, right? Mhm. It it it varies then because depending on what you're doing, you might want the max or the average. So it is a parameter that neural network people play with. But I found that max pooling is the one that people tend to always use. It just in practice seems to apply to most problems people have. So max means the maximum, right? Yep. Of all those in the set. I'm glad you said that because I thought max meant a male person and I just hate things being named after another male person. Or you thought it was like Maximilliam's score or something. Yeah, like Max's whatever. So, think of how do we apply that to like an image? Well, image is a a grid of pixels, right? So, you can make a little group, let's say a 3x3 square, and you look at those nine pixels, and you say, "Who has the most intensity here?" And then uh you move over by some length probably the length of the window of three pixels and then you know the 18 next door to that and you say what is the maximum intensity here and what do you think would be the resulting image if you did that to a photograph? I don't know. Um well for one thing it would get smaller right? What do you mean smaller? It's a little bit like down sampling. What is downsampling? So downsampling would be like taking a high-res image to a low res image. Well, high- res actually I don't know if there's a set definition, but people consider high-res something that you could print at. Oh, interesting. Okay. And what's low? There is not an actual like pixel or DPI amount. It's relative. And then relative to that, what's low resolution? Low res, I would just say like, oh, you only just use it on a screen. So, what's the difference between the two files? Well, the high-res one is bigger in size. Yep. There are certain cases when it would be nice to be smaller, right? Maybe you want to store a lot more photos compactly. You're just going to look at them on your phone. Or in the case of machine learning, you want to do dimensionality reduction. And that is one feature that pooling gives you. It takes this huge set of inputs, you know, all the pixels from the image and reduces them down by something. So if you pull nine pixels into one pixel and you just take the average or the max intensity, let's say, the resulting picture will be 1 nth the size. So I don't consider it down resing. I just say saving it at a lower resolution or converting it to a different format. Yeah. Yeah. Now in what ways does the image change by doing that? It gets smaller, but there's this trade-off, right? Yeah. Well, there's less data. And how does that manifest in the picture? You can't see as much. Could be like if I was holding a pen in a photo, you could zoom in and see the pen and be like, "Oh, it's a big pen." But if uh we converted it to a lower res, you might be able to tell it's a pen, but when you zoom in, you can't actually see the logo. Oh, that's a great example. Let's run with that. So, what if we were trying to build a system that detected if pens were in pictures, if people were writing things, and presumably we don't care what type of pen they are. So, uh you know, whether it's Bick or something else doesn't matter. Now, the fact that Bick is on there is almost kind of a distraction then, right? It's a it's features of the image that could distract the machine learning algorithm. It could be trying to learn, you know, how that logo looks in the input space when in actuality as long as you can detect that a pen is there, that's all you really care about. Mhm. So if you were to do pooling and in this case just you know down sampling the image basically or as you want to call it lowering the resolution or whatever that would maybe obfuscate the fact that the label the bick label is there but you would still see this this you know blue rectangle basically and then maybe an algorithm would have an easier time learning that a blue rectangle of a certain width and height ratio exists rather than the more detailed image. Let's take a break from our show to talk about the company Forbes magazine says is one of the top 10 edtech companies you should keep an eye on. I'm talking about Data Society at dataocciety.co. Data Society is an online training program for professionals. I took a look at their curriculum and one of my favorite parts is their use of realworld data when they teach you techniques. I mean, there's nothing wrong with the Irish data set or Titanic and theory, but wouldn't you rather learn on more practical applied use cases? Their courses altogether have over 20,000 lines of code spread across tons of examples, and they're all downloadable materials. No successful data scientist starts from a blank file. We're always building off good boilerplate code, and Data Society is full of that reusable, downloadable code, all with printable reference decks and supportive forums. Head over to datasocciety.co/dataskeepic and use the promo code skeptic 15 for 15% off any membership. I want you to check out their award-winning platform, comprehensive data science courses, and curated code. Do that at datasocciety.co/dataskeepic and use the code skeptic 15. The other reason I like the pen example is because a pen is very thin, right? It would be easy for the pen to even disappear if you downresed by a lot. Like let's say you went from you went to a tenth the size of an image. You could see where in that reduction the pen might just get erased because when you shrink a pixel or a group of pixels into one, the pen contributes a small amount to the average. Right now the max might distort the image to like what we like aesthetically as human beings. But the results it could have would be these lower level features might say, "Aha, I think I see a pen here." And if just one or two or in the case of MAC pooling if just one neuron emits a high output saying I think I'm detecting a pen or some feature that is you know seems to be a pen then when you say all right anyone in this region these nine pixels if even or nine groups of neurons or neuron groups or whatever if any even one of them thinks it confidently sees a pen consider the whole area to contain a pen. Mhm. And in that way, you're doing dimensionality reduction, but you're also magnifying some of the like learned features of the network because you're amplifying them a little bit. Since you are losing information, you're going to a smaller size. It can also help prevent overfitting because you're sort of voluntarily, so to speak, throwing away data. So, it would be harder for the algorithm to overfit since, you know, it's losing information at that step. pooling actually shows up in pretty much all neural networks, but especially the convolutional neural networks we talked about last time. Do you remember how we described a similar thing where there'd be a grid and it would shift around in what we commonly call a convolutional layer? Usually there might be first that step where it the convolutional part tries to learn some new features of the data and then the pooling comes in and says all right these neurons they you know computed something now I'm going to take the max value of that group and then pass that forward in the network. So the next layer of the network it sees like an aggregated this max pulled representation of the outputs of the previous layer but it also shrinks them down. So now the learning space is smaller. So it doesn't have to learn on the whole image. It learns on this reduced size that hopefully is emphasizing the features that are useful for any kind of prediction or classification you want to do. So most people use this for images. Certainly. Yeah. I can't imagine making uh a neural network on images that didn't use pooling and still computed in a reasonable amount of time. But it's also relevant in like audio analysis. Some of the stuff I've been doing recently that we're going to talk about in a couple weeks on the show. And yeah, there's pretty much pooling in every use of deep learning, I would say. So, at least the way I interpret it, it's like pooling as kind of like, oh, if present, round up. If not, round down or something or like kind of rounding to black and white. Yeah, in a way that is a pretty good analogy. I could pick it apart if we were being really technical, but if if that's what you walk away with, that's a decent understanding. That's what I walk away with, so I'm sticking with it. Fair enough. that works for me. Well, my takeaways are slightly different. For me, it's that pooling can help your network in a lot of different aspects. For one thing, it definitely performs dimensionality reduction. That's the biggest gain in my opinion. Secondly, it helps a little bit to regularize your network, which is a fancy way of saying prevent overfitting more or less. Third, or maybe this should be second place, depending on the lower level feature engineering, it can help amplify signals that might get lost. So in the same way when we change the resolution on certain images, things like chain link fences can become like really weird looking and hard to see, you don't want to lose information that you might lose if you took the average in a pooling step. The max pooling step can sometimes help you capture important features that the lower level networks have detected that you no longer need to know as much about the full resolution of your input data. You just can look at now that the it was aggregated into at least one of them in that group found some important thing and then learn you know when that neuron lights up how much does that correlate with your final output. So anyway that's pooling pretty simple concept but uh thanks as always for joining me Linda. Thank you Kyle and until next time I want to remind everyone to keep thinking skeptically of and with data. Data Skeptic is a listenerup supported program. To support the show, visit dataskeepic.com and click on the membership tab.