10: Mid-Level Vision

Key Takeaways

Let me just kick things off by showing you a picture. So this is what is called a flat map of the macac brain. Okay. So it's the so the the the cortex is this sheet, right? which in uh your your head is kind of folded up into this the shape of your brain basically, right? Um and a flat map consists of taking the cortex, making a few cuts and then kind of flattening things out. Okay. Such that you can kind of see the entire cortical surface at once. All right. Um and this is a flat map um where where the different what are considered to be different regions of the visual system um are outlined um and different areas are in different colors with the all of the regions here that are colored are considered to be part um of the visual uh visual system, right? That means that they respond when people are looking at stuff, right? And so this is like the intact macac brain. And so you can see that the the visual system is kind of like the back half of the brain along with some stuff here in the frontal lobe. Um, and this is kind of what it looks like when you flatten it out. Okay. And so there's like a lot there's a lot of different regions and it's a big chunk of the brain. All right. So what is it that distinguishes the different visual areas? Well, um, the the main criteria by which you typically distinguish visual areas um is retinatopy. All right, where each visual area typically contains a retinopic map. All right, so a map of the entire visual field. Um and so these the visual areas can be localized by measuring retinatopy. Um in this case by measuring how the neural responses change with stimulus location. So this is um an example where a participant in an experiment is looking um at this display. So they stare at the fixation point and there are these um annuluses that kind of gradually expand. So they're checkerboards which are these high contrast stimula which produce big visual responses. Okay. So the idea is that it expands and so the location of the visual stimulus um varies o o over eccentricity uh as a function of time and in this case you have a pie wedge of a checkerboard that rotates around. So you kind of map out polar angle and so in each case every point in the visual cortex is colorcoded in this case as a function of the eccentricity uh at which the stimulus evokes the biggest response. And so you can see that you get this kind of gradient here. Um and in this case as a function of polar angle and you get another gradient that's kind of orthogonal to this. Okay. So this is the retinattopic map that we have seen before. So we talked a lot about this in the context of primary visual cortex. So remember um this diagram where um this describes how the two visual fields get mapped onto your visual system. So each eye can see part of both visual fields. Um but then um the contrateral visual field kind of gets so or I should say the the left visual field gets mapped to the right hemisphere and the right visual field gets mapped to the left hemisphere. So this is an example of an experiment where you can kind of see a whole bunch of different visual regions. So this is one he this is from one hemisphere. Um and and here the color is mapping out the the polar angle at which the stimulus evoke the biggest response. And so you can see in what's considered to be area V1 primary visual cortex um the color kind of sweeps from green to red all the way up to dark blue. All right. And so this is this is one hemisphere. And so you're only getting um one hemif field, but you see the full kind of range of visual angles. So then what happens um once you kind of get to what's what's considered to be the border of V2 is you kind of start to see a lot of blue and green, right? So there these gradients, but they're they're blue and green and there's not a whole lot of red. Whereas below V1, you see gradients from red to green. Okay? Um and so this is an indication that there's kind of a separation um of the upper and the lower hemields. Um and there's but they're inverted. So the lower hemield is mapped on onto the region above V1 and the upper hemfield is mapped onto the region below V1. Okay. Um and so these visual areas have been given names um that are a little bit complicated sometimes because of the sort of nonlinear his historical trajectory over which they were discovered. But you've got V1, V2, V3, and then there's this other one called V3A. Um because they'd already discovered V4, and then they they realized there was this extra region and they had to squeeze it in. Okay, so there's stuff like that. All right, so there's lots of these regions. Key idea um is that we distinguish these different regions via retinatopy, right? And so in particular, what happens when you get to the border of a visual region is that the sign of the retinatopy flips. So here it's going from green to red and then it switches back to going from red to blue. Okay. And the same thing happens with tonotopy um in the auditory system where you see a reversal of the sign of the map. And so you know in the early days like long before there was fri and you could look at the stuff in humans, people were doing this in non-human animals with electrodes. So they'd have electrodes in uh in the brain. they'd be measuring receptive field locations and moving the electrode around and observing how the location of the receptive fields kind of varied across the cortical surface. But the same kind of principles applied. It's just with fMRI you can kind of look at like everything at once. And so it's it's helpful for looking at these maps in this way. Okay. Okay. So, another kind of um important concept that we've already sort of alluded to um is this idea that there are are you can think about the visual system as consisting of pathways um and that these pathways kind of have their roots in the retina. All right, so we've talked about how you've got um cells and par parasol cells um and that they project to these different parts of the LGN. Now, okay, this diagram has this other slightly annoying complexity to this, which is that occasionally um people change the nomenclature with which retinal ganglen cells are described. And so sometimes, um the parasol cells are referred to as M cells. Um you know, I guess they thought this would make it easier to remember that they project to the magnellular layers. Um and sometimes the cells are referred to as P cells. Okay. So, it's it's just a gigantic mess, but you'll you'll all keep it straight. But the essential idea, right, is that you have these two different populations of neurons in the retina. They have different properties. Um, and they project to different layers of the LGN. Those in turn project to different parts of primary visual cortex, these different sub layers. Okay? And then those in turn project to different parts of subsequent areas. Okay? So area 18 which is also often known as V2 um has these different subcomponents to it and so there's this degree of functional segregation that kind of persists um and it persists even up to kind of um areas that are fairly deep in the in the hierarchy. So um inferior temporal cortex and the the pietal lobe. And so um one of the you know one way to kind of think about this organization that um remains very common and probably to some extent true is this idea that you can think of two main pathways in the visual system. One that extends ventrally that mediates object recognition culminating in infratempaloral cortex. It's often called the what pathway and then one extending dorsally culminating in the parietal lobe um that's involved in the localization of objects often for the purpose of mediating actions. Okay. Um but to some extent this um division kind of has it has its roots kind of earlier in the visual system. Okay. So this is a really famous picture um kind of looks like a a horrible subway map but it's actually a diagram of of the primate visual system. So each little box on this picture is a visual area. So distinguished by retinatopy and then the lines between them represent connections or or I guess sort of fairly dense connections dense enough that they considered it worth plotting on the graph. Okay. So these are the retinal ganglen cells. Um you've got these two classes predominantly the two different layers types of layers of the LGN. Um V1 V2 um and then an assortment of other areas. Um, and so you can see that the things to take away from this are one, there's a lot of areas. Two, there's a lot of connections. Three, these are organized and conceptualized in a hierarchy. So, we kind of start out at the beginning of the system where the light enters. Okay? And then some of these regions are kind of situated more more deeply into the system than others. Okay? Okay. So that's like a very important um uh idea is that the sensory system is hierarchical. So there are some regions are getting input from others and then providing input um to others. And if you look at what happens when you move deep more deeply into the system. So if you if you record from neurons in these different areas and try to understand what they represent or what they're responding to. Again, another common theme is that as you move more deep into the system, the responses get more complicated. All right? Um, and so there are very famous examples. So famous that you've probably encountered them in other classes. So this is an example of a neuron in infertal cortex um that's selective for faces. Um, so it will respond a lot when when you present it with an image of a face. Um, not when you less less so when you blur out the eyes. Um, it can even be kind of a a schematic of a face and you get a pretty good size response. Um, so these neurons are much harder much much more difficult to describe mathematically than the neurons that you see in V1, but often selective to like more complicated things that are more behaviorally relevant. And so there are some pretty famous examples at this point um that really illustrate this point. So how many people have have heard of the Jennifer Aniston neuron? Yeah. So that's sort of one that made its way into pop culture. Um so this is in infrate tempmporal cortex of a human who evidently you know watched some TV or movies um and you can see so the graph so the the diagram here shows example images and then underneath it um the response of this particular cell and so you can see that there are these two different images of Jennifer Aniston they both produce a pretty good size response um this is kind of non-trivial because like the images if you look at like the pixel intensities like they don't really have a whole lot in common, right? Even in abstract terms, I mean, there's glasses being worn in one case and not the other. Um, there's just lots of differences. Um, and it doesn't respond to like lots of other images. Looks like, you know, Brad Pitt decreases the response a little bit. So, um, you see stuff like this deep in the system. Um, this is my favorite example. Um, this is a neuron in entrinal cortex. I don't know how easy this is to see, but this is somebody who evidently liked Star Wars because the neuron it uh responds to images of Luke Skywalker, but it also responds to um uh the words Luke Skywalker in text. It also responds to someone saying the words Luke Skywalker. Um and there's a moderate response to Yoda. Um so you move deep into these sensory systems and um things get more complicated. That's a common theme and we'll and we'll elaborate on that in in much more um detail and much more rigor uh in subsequent lectures. Um and um a lot of times like one one of the other things that you often see see signs of as you move more um deeply into sensory systems is that the responses become more invariant. Um and remember at the very start of the course we talked about one of the challenges of of in particular recognition tasks. So, one of the things our sensory systems are good for is helping us recognize things. And one of the challenges of recognition tasks um is that different images or different sounds um of the same thing in the world can often physically be really different, right? And so these are all different images of a house. Um there's in fact an image of the same house taken from two different viewpoints. So you can both tell that these things are all houses. You can also pick out which two are the exact same house. But of course the pixel intensities are completely different in all of these different cases. Right? So somehow or another one you have to construct uh representations where uh those invariances exist now. What we're going to talk about today is not this kind of stuff so much but something that's kind of in between um these very complicated recognition tasks and what we call early vision. And and this normally is what is referred to as mid-level vision. So remember that the big picture here is that perception involves inferring the structure of the world from measurements of energy that that are generated by the world. Right? So in the case of vision, these are patterns of light. And so far the past three or four lectures we've been talking about what is often referred to as early vision. So early vision we we often think of as like a set of useful measurements. Okay? So you get this image as an input to the retina. Um and then there are a set of filters um that are measuring different things about um the image in different positions in space. So in V1 for instance the various kinds of receptive fields give us local measurements of orientation, contrast, disparity, color, spatial frequency and so forth. Sort of like the ingredients of of images. Now there are also perceptual phenomena that are linked to these measurements that also kind of fall under under what's called early vision. And those tend to be phenomena that people classically think are explained by those measurements. So for instance, we talked about these adaptation effects like the tilt after effect, right? Or spatial frequency adaptation and the effect on the contrast sensitivity function. Okay? So there are perceptual phenomena that fall under the rubric of early vision. Um they often are used to to make inferences about the mechanisms of early vision. Okay. So mid-level vision um by contrast typically refers to processing stages that involve inferences uh that in some way are about the world that are based on measurements that that are made made um in early vision uh eventually leading up to object recognition and scene perception which typically people would call high level vision. Right? So we got early vision, mid-level vision and high level vision. These are sort of loose and sloppy terms, but they're nonetheless kind of used to sort of indicate kind of different parts of the field and different aspects of visual perception. And so and the other thing that I should say is that um unlike early vision where there's often like you know fairly reasonable linkages between some of the associated perceptual phenomena and neurons typically retina LGN V1 um mid-level vision and the the various perceptual phenomena that are associated with it um is less well linked to specific anatomical stages of the visual system and to individual neurons. And so in some cases those relationships exist and we'll talk about them but they're they're not as tight. Okay. So one kind of um important theme that's sort of central to mid-level vision is the idea that local measurements are ambiguous. So neurons in in the visual system in some sense view the world through these little apertures, right? They have receptive fields. Okay? So that means that there's like a region of visual space that drives their response. So they're looking at the world through these these localized spatial receptive fields may you know measuring something that sort of happens here or happens here or happens here. Okay. And these local measurements um are are often quite ambiguous. So they can be ambiguous in terms of what is actually causing the image intensities that that's with within that local region that's being measured. So this is a a a image that was designed by my PhD advisor Ted Aden and it's um it depicts sort of a a simple thing that like looks like it's painted different colors. Um and so you have these different kinds of edges. So you have this edge here and this edge here. And inside those apertures, the image is exactly the same in the two cases, right? There's kind of light gray on one side and dark gray on on the other side. But you can kind of just tell from looking at it um that this is like a change in pigmentation, right? It looks like a change maybe a change in the color of paint that was used to paint it. Whereas this is due to what's called shading, right? The fact that there's different image intensities there is because the surface orientation changes, right? So of course, you know, you look at this thing and your visual system acting in concert is able to come up with the correct interpretation. But the point is that the local measurements that are being made early in your visual system are ambiguous, right? And an individual local measurement on its own is not enough to tell you what's actually happening in the world to cause um that that particular pattern. You can see similar kinds of things in actual um images. Um, so these are some interesting cases where so each of these patches here, this one, this one, and this one um are kind of close-ups of three regions of the edge of this log. Okay, so there's just there's just a log on a background of stones. And again, when you kind of look at the image, it's sort of obvious that there's the edge of a log there, right? But then if you zoom in um to what's actually evident kind of at the level of like a receptive field um you can see that it's it's really not very clear. So in in partic like like this case for instance, right? If you look closely like the edge of the log is kind of like right here, right? But in fact all the contrast in this particular local region is like up here because there's a shadow being cast by a rock that's kind of right next to that edge, right? Um, and so the point being that if if all you had to to analyze was this and you were trying to figure out where the object boundary was, it'd be pretty hard to do, right? Um, and you know, some the regions vary in terms of like how how much evidence there is for that edge. Um, but they all have some degree of ambiguity. Okay. Um, and that's actually like um a really useful exercise to do if you really, you know, want to sort of under under appreciate this u more. You can kind of take some take some images and view them through a little aperture, right? And if you look around with the aperture, it's like really hard to make sense of what's going on. Okay, so local measurements are are ambiguous. in order to make inferences about the world sometime somehow they have to be combined in some way. Um and so one kind of phenomenon that that has uh some relationship to that general idea um and is typically associated with mid-level vision is perceptual grouping. Okay. So this refers to the fact that things that are similar tend to subjectively group together, right? In the sense that they they appear to be part of the same thing. Um and there was a movement in psychology long time ago called just psychology and one of the main things they were interested in were these grouping rules. Okay. So what is this all about? So if you look at these two images there's a sense in which like you see these images as consisting of rows, right? Like you kind of look at them and like the natural description is that like you have rows of circles and squares or black dots and green dots. Okay? And so that subjective sense that you know these circles kind of belong together and these circles belong together is called grouping. And in general um similarity is a major factor that kind of determines grouping. Um common fate. So the fact that things move together that has a really big effect. So that thing that kind of looks like a number four they all look like they're part of the same thing. There's texture grouping. It's grouping by proximity. So again, you kind of you look at the thing on the left and you see these horizontal rows presumably because the the circles are kind of closer together in the horizontal dimension and the vertical dimension. You look here um and you tend to see this as two groups of dots. All right? So there's lots of different ways to kind of think about these sorts of phenomena. You remember this idea that there are these different levels of analysis. There's the computational level where we can talk about the problem that's being solved and the constraints that allow it to to be solved. Um there's the algorithmic level. There's the implementation level where we talk about how you would describe things in terms of neural circuitry. Um and it's possible to kind of think about grouping at all of these different levels, you know. So for instance, we can give an explanation of grouping in terms of uh of neurons. for instance, by saying that there are neurons in the brain that respond more strongly when their neighbors, which will have nearby receptive fields, are also responding. And in fact, we'll we'll sort of see some evidence that there's probably some truth to that. Um, and so you get a strong response when many dots are in a line or in a clump, right? So that's one way to kind of explain how the perceptual effect of grouping would come about from neurons. Um, but it doesn't really tell us why that would happen, right? So alternatively we could talk about grouping um in terms of probability. So the so the Helm Holtzian approach to thinking about this sort of at the computational level would be the idea that what we see as our best guess as to what is in the world based on the input data and based on our prior experience. So you might suppose that when things are close to each other in the image, there's a good chance that they're actually part of the same object in the world. Um and so we have a tendency to see them as part of the same object because we're sort of that that sense that we see them as the same thing represents an inference um that they're actually caused by a single object in the world. Okay. So the idea here is that there are these heruristics um that are based on probabilities in the world um and that's a perfectly useful way to think about this phenomena but it doesn't tell us anything about like how you would implement this with neurons. You have a question? Yeah, this uh so I saw like um like a posture outside your office something along the lines of testing these illusions with uh like like legally blind kids who've like just like had surgery gotten their vision back. Um >> so if if this household's explanation actually is true um does this not work? Does what not work? >> Do like do would like a like a person that like just got like like their vision from like a surgery see the the right side for example like >> Yeah. So the so the so the question is like you know would would someone who's been you know visually impaired or or or blind from birth who suddenly has their vision restored would these effects hold for them? Right. And I don't know um I mean something that could be tested but I think you know the it really just depends on the extent to which all the all these factors are really something that result from evolution or result from development. Um and it could be some combination of both. Right? So one possibility is that your brain is wired up in a certain way in order to leverage um these probabilistic relationships that exist in the world. All right, so natural selection sort of caused um the brain to evolve to give rise to this kind of grouping. Okay, that's possible, right? And the prediction there would be that well from birth or maybe largely independent of developmental experience, you would you would experience this more or less the same, right? Um but the other possibility is that these are things that we learn online um over the course of development just by our interactions with the world, right? By by looking out and and then grabbing things and realizing what actually constitutes an object and stuff, right? Um and in general, we we we tip there's t we typically don't know a whole lot about the the relative importance of development and evolution for a lot of things in perception, right? So a lot of those questions are still um not very well answered. And so in the case of grouping uh I don't really know the answer to that. And another another way to test that would be to you know ask well you know what would happen if somebody was reared in a world that had very different statistics right um you know would they end up having very different perception? You know that's another way you could in principle um ask that question. Of course that that that natural experiment doesn't typically occur. Um, and you know there's no there's no way to do that in people often, but you know sometimes you can do those experiments with with animals in the lab and you might in principle be able to do that. Okay. Okay. So this is perceptual grouping and there's these different levels of explanation and there's lots of factors that affect grouping. So this is what's known as grouping by good continuation. So you see an image like this um and what you perceive is a circle and a square, right? Um, now this is a perfectly valid description of how you would create that image. You got a Pac-Man and then this other funny shape, right, that are just placed together a certain way, but that's not what you see, right? So, what you tend to see are things that are kind of continuous contours. You also group by closure. So, this this image on the right um could be described in a whole bunch of different ways. For instance, you could see it mostly as these sort of closed shapes. You could see it as these kind of cross-like shapes. You could see it as I don't know what that what you call that and funny like X- like thing right okay they're all valid descriptions of the image but people tend to see the circular organization right so and again the the naive idea is that like well closed contours are pretty common in the world that objects tend to be like that um and so your visual system kind of is is wired up or programmed in some way to enable you to see that okay I want to end there when we come back um we will resume talking about uh grouping. Um have a good weekend. Um and actually it's spring break, right? Okay. So, um I hope everybody has a great spring break. You get a chance to kick back a little bit. Um and I will see you in like 10 days. Bye.