Exploring “Falcon Perception” with Yasser and Phúc

Key Takeaways

Like always, give me a few seconds to get set up. Oh, here. Uh this is working my Is this working? I'm not sure it's connected though. I It is connected. Perfect. Okay. Here. Good. Oh, we have somebody in our Uh we both got a bill by the way working from Algeria for the dream of being an AI engineer at the Apple. Wish me luck. Hey, it's funny because I met the in San Francisco an actual ML engineer uh that worked at Apple. Um Yeah, so if you're listening to this, shout out. Uh thank you for um uh the dinner that you bought me when I I was there. Okay, so this is good to go. And uh I'm pretty sure there must be a better way for me to kind of get this whole thing set up. Uh okay, wait. So I have to go there. This one is working also. And then last one is this little guy. Okay. And there we go. Uh by the way, um I'm the one of the first author of this paper. Yes, I is actually from Algeria. >> [laughter] >> So uh yeah, we we have that covered. Um Cool, guys. So like it Today what is a this a little paper. Let me show it to you. It's this one. Here we go. So it's a Falcon Perception a paper. Um well, I say paper, but it's like actually um it's a collection of three paper. So you have the Falcon uh perception slash OCR. You have the Sign uh know um kind of distillation method that they using for like initially using their weight. And then you have like the PB P bench uh over here that they're using to kind of test uh this stuff. So like there's actually three paper that are related here. Um yeah, and then like um basically the whole idea of this is that um instead of having like um uh like the you know for like vision uh dense kind of vision task, uh instead of having like the decoder encoder encoder decoder paradigm where you train the encoder and then then you like kind of lock it with a uh with the decoder, you they're trying to break this paradigm and then blend it blend this whole thing into one dense um transformer. Uh so it's like 22 to 26 um layers here and basically like the um the image element is going to look at every patch will look at every patch. And then the language element that is just like literally next to it will look causally at um at the tokens. Uh and then you get like this this this stuff, right, at the end. And based on this and like a few uh very light um uh custom head here and an image feature up sampler, you can get like something similar to Centry. Um much smaller and actually really performant, especially in some specific um uh like benchmark that they're doing for perception. Um so like that's kind of what we're going to look at. Um we have the chance of having the two of the authors here um uh from TI and we're going to be able to answer like ask them a whole bunch of question. Um yeah, I've prepared quite a lot so we'll be able to uh chat it out. If you have any questions, um don't hesitate to just like shoot them uh in the chat and I'll try to kind of weave them in as we go through this uh with my guys. So uh I'm going to let them in here. There we go. And okay. And sorry for the voice. I'm I'm recovering. I I was a bit sick. Uh but now I'm getting a bit better. Um Yeah. Uh but uh I I'm at like 56% uh of my brain power. So this is good and I have all my questions right here. It's going to be a fun one. >> [snorts] >> So they're getting there. I like this paper because like um it's kind of gigantic, but it's not too daunting. Like um uh all the how would I say it? Like all the all the different sections kind of built upon each other. Um and they took a lot of time to be able to um like uh walk you through like how they're assessing that this thing is correct. There's a lot of ablation throughout. That's a really good one. Hey. Are you Can you hear me? Hi. Okay, perfect. >> Yes, can you hear me? Yes, yes, yes. Uh we're just going to wait for Yasser and then we should be good to go. Uh where are you guys located by the way? We are in Abu Dhabi. Okay. It's good. I I can hear the delay because like whenever I have a call with someone like that far off, there's usually like a 2-second delay between like what I'm saying and then like the people understanding it. Yeah, this is good. Um You were in Canada? Yeah, I'm uh just uh north south of uh Montreal in Canada. Um So for me it's just it's just noon right now. I think for you it's like kind of evening, right? Uh yes, 5:00 p.m. over here. Yeah, yeah. Uh and uh I think Yasser Oh, I think he's coming, but he's having some issues. Hey. Uh can you hear me? Uh we can't hear you, but uh we see you. We're going to We're going to get there, folks. We're going to get there. Uh by the way, how do you pronounce your name? Uh uh My name is Yacine Okay. Yes. >> [snorts] >> Uh that's perfect. And Yacine. Yacine, yes. Is that correct? Okay. Yeah, that's good. It's good enough. Okay. I think uh Oh. Okay, he just left. Um I think it's has to do with his mic or something. Okay, round two. Can you hear us? >> [laughter] >> I think he Uh he dropped again. There must be something um that uh Oh. Can you hear us, Yasser? Yeah, we can hear you, but wait. You can hear us, but we can't hear you. Somebody's asking >> Somebody's saying hi to me in the chat. Yeah. They're saying hi to you on YouTube. >> Hello. Uh somebody asking est-ce que tu es d'origine française? I'm uh from Quebec, actually. Um and we do speak French over here. So, I do have an accent. Um if this is what you're you're asking. Um No, we can't Did Chrome ask you for permission? Did you give permission? I think he's going to uh drop off and come back again. Maybe we can start with the an introduction about the your background. Um I got to do a bit of my research, but like it would be nice for everybody to kind of listen from your point of view, kind of your research background coming to to this work. Yeah, so I did my PhD in computer vision in Ireland from 2019. Uh actually me and Yasser belong to the same cohort, so but yeah, my like my my my interest uh has always been on around like representation learning especially from computer vision and back then uh self-supervised learning, scaling for both text and and vision were like very big deal back then. And of course we know already for NLP the one tool uh objective is just next token prediction. >> Yeah. But for like computer vision, it wasn't really obvious back then. And and even now in in in some sense, I I I still say. But there there is a lot of uh research back then. And uh uh my especially my focus was on contrastive learning. But then I uh then I I I um I realized that doing research at that level, especially representation learning, proving it in real world they require a lot of resource which is not really available to me. And so I kind of pivot a little bit into a niche object uh subject which we call the second object-centric representation learning. Okay. Which is in contrast to like learning visual representation of an image. And here we we try to learn and decompose from the image uh uh the the set of representation which corresponding to the object itself. So, we can capture the the property of the object in the latent representation. So, this kind of object-centric representation learning, they were like uh uh sup like people hypothesize this would be like uh like like the unit for reasoning, visual reasoning. Mhm. If if like if follow Jan LeCun, he he he he keep talking like this like uh representation vector is the is the unit and uh and and the algebra will be the logic. So, that's what kind of the the promise but the the field was still very early back then. We we train on synthetic image and the object align simple geometric object. Yeah, but then uh but then so that's the PhD. The PhD you you allow to explore Yeah. uh on the niche on the on your interesting topic, but then end of the PhD, you know, the the field move on. Yeah. ability grow everything focus on LLM. Vision or you also build on top of LLM. So, so yeah, here we are building uh a field uh Good, this is really cool. And I like one thing I like about this paper is that I could see um you guys try to like uh morph the the the the problem into something that like makes sense and um what's it have the similar like LM shape right? Where like you have like this unit and then you can kind of iterate on um without having too much like biases and and and load. Um It was really cool, honestly. I think I think I I really see like this kind of progression from what you you were saying to uh what you guys were able to do. Um Oh, uh Yasser, are you can you hear us? Yes, can you can you talk? Do you have a mic working? Not yet. Um is it uh No, there's no yeah, there is an issue with your mic. Um because we can't hear you at all. Uh no, yeah. Uh wait, um there's a question that I think it's for you. Uh what do you think about François Chollet approach and his kaleidoscope uh kaleidoscope hypothesis? I'm not sure what this is about uh the kaleidoscope hypothesis. Um Do you know? >> Yes. No, I know I know François Chollet and his uh RAGI benchmark. You're right. Yeah, yeah, but uh I'm not really sure what it is uh kaleidoscope Okay, I think what he what it's saying is that like while the world appear of well all of a small number of core repeating pattern atoms of meaning that are combined in different ways. I think like that's kind of like the unit that you were talking about which is like >> Yes, yes. if you I mean in language we see that like very clearly. Like you can like kind of have just by doing like next token prediction, you're able to kind of learn this meaning. Um but I think you're trying [snorts] to get to the same thing for in the vision domain. Um Yes, I yeah, I think it's on board to the like the same thing like I think as Plato said, you're trying to carve the nature at its joints. Trying to factor out all the complex reality into its independent component and and decompose from there. Yeah, yeah, yeah, yeah. Yeah, exactly. And then you just kind of feed the model with high-quality data and you let it learn, right? Like that's kind of the dream. Um oh, there's somebody in Twitter that uh says, "I love Falcon perception model, it's a great model." Um So, kind of my saying this uh I think it's for you. And um how how close do you think um we are from from that like find this unit for vision? Uh I think we are I think maybe we already there. The unit is on the it's just not interpretable for some reason. >> Yeah. But like uh uh maybe while waiting for Yasser, I can have a story about a bit of election. So, so like I will mention you about during my PhD about this object-centric representation learning. But then around the end of the PhD Can you hear me now? Okay, good. Now we can hear you, yes. I just connected from my phone. I don't know what's wrong with the um the laptop though. Ah, it works, it works. It doesn't matter. Thanks. Thanks for being here. I'm connecting from my like I just downloaded the app. And that's it. Sorry about that. I'm sorry. You can continue and then Yasser, you can maybe introduce yourself after we Sure, yes. But like I was saying like for the for the vision encoder back then, there is a paper about register token. Okay. Where where instead of in the vision encoder, you only have one token for each patch, now they are appending a few extra token which they call the register. And so, the purpose is just for the model to put uh any unused attention or any extra information, global information, into the register token and doesn't disturb the the representation of the of the patches. So, that they can keep a very good like uh special feature in in in the token path. But then when they look into the register token, the token actually capture uh semantic uh high-level semantic and corresponding to individual object, salient object in the scene itself. So So So in a sense, given enough capacity inductive bias in the current model, we we will we might already have something like that internally in in in in vision encoder. Right. Cool. Um Um that's great. Thanks uh for for uh this intro. Um Yasser, maybe you can introduce yourself um and your background to um uh to the community. Yes. Um so my name is Yasser. I am a uh researcher currently at TII in Abu Dhabi. Um I've done my PhD in DCU like uh Dublin, Ireland. And my main research back in the day was focused on self-supervised methods for um for images. Like how do we learn without having access to labels? And um like upon joining TII, I started working on large like mostly large large language models um starting from Falcon 1 um 7B for 40B and what 8B like um a year ago. And then obviously because my background is in computer vision, so with the with coming up of like visual language models, so currently mostly working on uh on this topic. Uh yeah. Cool. Very nice. And um you were both at like uh Dublin uh uh City University at the same time. Um did you start to collaborate there or like um only when you you guys moved into um um the IIT? Yeah. Uh not really back then. Okay. Okay. So Yasser actually come here first and then uh Yeah. Uh And then you collaborated? Yes. So Fu used to sit like uh besides me in the office in in in in Dublin. And we were in in uh we were mostly like doing like you know paper reading um together like maybe twice a week. So this is how we started like we were like analyzing papers and whatnot. But not really like working on a common project. Although we had like you know a common supervisor. Cool. And then you decide to bring him in uh in the team uh later on, right? Yes. Good. Um this is fantastic, guys. Um Um can you uh explain us like the vision behind like um Falcon Perception? Because I think like the just the philosophy of it uh is very educative about like why you made certain um architectural decision, maybe. Um so Yasser, maybe you can explain to us like um uh what's the what's the the grand idea uh behind uh this model? Yeah, so like essentially um like if you are doing um let's say for example this specific task like dense grounding or open vocab segmentation slash referring expression segmentation, um like like there are like a lot of models of like for example at least some or DETR architectures um which do similar stuff. But for us, we are like mostly like kind of exploring like a different like scaling path uh driven by a driven like a with a different like end goal. So if the end goal is simply to do I would say simple nouns for like open vocab segmentation like segment this car, segment this headphone, this bottle and whatnot, I think like these DETR architectures might be actually good enough. But for us, like the long-term goal was and is to kind of build this what you call like a perception engine um that either a larger model can call or people can interact with um like in a VQA mode. And this model, it has obviously to do like you know the current like task which is like open vocab segmentation. So or or or or or stuff like this, but also captioning or what you call like grounded captioning. This image has this person, but it has to know where exactly this person is. Um we need to have a good what you call like OCR as an attribute, which is like our level two in P bands, which is like identifying things uh using um like what's written on top of them. Also like spatial constraints like this person on the left, this person on the right, uh behind and whatnot. Um but also we care about dense scenes, which is like let's say when we have like scene with 200, 300 even like 500 instances of the same object. So this is the second I would say requirements. The third one would be we want this model to act in a in an agentic way, which means given an image, it has to be like zoom in, crop, apply some um apply some operation if needed to to give the uh to give the answer. But also we want this model to have like to be able to know the limits of perception. Say if something is unclear in the image and whatnot, like some text is blurry, it has to tell the um like the user or the model calling it like something is like unclear, can you take another picture or something? Right. So this is that was like the design we wanted to have in this model, right? And once we committed to these goals, uh which means like for the first two, we need like you know the model to understand the problems very well and long long-tail concepts. And to do this, like for the model to understand like you know the queries and what people ask, people can like you know like make typos, can ask in a very I would say unclear manner. There's a lot of intent behind it. So obviously we need to mix in text data because this is where the knowledge comes in from. Um if you want like longer tail concepts, like you want to discover more, you need to train on captioning. So because captioning is cheap at large scale, which means you can know a lot of concepts from uh from captioning. Interleave documents, which mean and and obviously the current task, which is like open vocab segmentation. And essentially the design the design behind the model like because we want all of this, it has to be like you know a single like sequence model. We cannot go for like design all the losses and all the architecture to fit only open vocab segmentation because this is this is one of the goals, but not the the majority of the goals. Um so this is why why we have done like auto aggressive in the first place. Um because NTP gives us like this uh I would say umbrella behind all this kind of stuff. Now, for for dense grounding specifically, um like architectures like DETR usually need what we call like multi-scale because you don't know at which layer the right scale is for every query or every object. So people like you know do FPN feature pyramid networks, so which means they feed um different uh image representations from different layers to the to the decoder, right? But we wanted this to kind of emerge as part of the architecture. That's why we do early fusion. Because if you do early fusion, which means like text and and and and and and images, they have access to each other all along the way. Which means for a certain query, the model can pick like the suitable layer in a learned manner. Like this is part of the objective we we we we we we minimize. So this is why we have we do like uh like early fusion. And one of the reasons also is why we wanted this kind of I would say simple interface uh in an auto aggressive way is because we want also to attach ourselves to what people are doing in in LLMs. Um because in LLMs obviously like you have like a lot of scaling laws, uh a lot of tricks on how to do like long context sampling, like RL and post training. Um so essentially like you know by doing but like Falcon Perception V1, we can like already like borrow a lot of tricks from the LLM community uh for free. So I would say I would say Falcon Perception V1 on the current version is is the first step. It's like step number one. Uh today it does like open vocab and referring expression segmentation. But the point is we want we wanted to lock this architecture first and then we can add capabilities on top of it without uh without kind of redesigning the system. So that was essentially the goal like behind the the choices uh you see in the architecture. Yeah. Kind kind of like uh uh let's say like um having uh like the this first I don't know like the uh the decoder version of it, right? Uh for language where it was if you look back right now like it was very naive, right? But just scaling it led to great things and then like you can uh tweak the internal whenever you want to have more capability and then you rescale it and then you rescale it again and you have these waves um that led to more capability. The one thing that I um I'm trying to like understand um uh why do you think what's the intuition behind like um uh uh like this being something that can be called by another um language model or or or something like that uh versus uh it being literally part of like the uh uh a bigger model. Um what's the intuition here? Like you mean to say why this has to be a sub as as compared to like itself having doing the entire stack of probabilities. Yeah, exactly. Because like you're you're still doing um language in this. Like I I understand like it says 600 million parameters. Of course, it won't be able to do all the stuff. But you do think that like this paradigm if you were to let's say scale it to the size of the stuff that we have right now, um will end up overtaking this or like you think it's still going to be something fundamentally different and that will be like it's all one core unit. Um to be honest like like I don't really have the answer to this question now. This is what we are doing currently, what we are exploring. Um I would say like yeah, essentially the question is do we need to take like this large orchestrator or large like VLM that has kind of better knowledge um like a reason over like tools and what's not as compared to like using using Falcon perception that say V1 as a like just to do ground. Um so this kind of division of labor is not fully settled in my opinion. Um I see I would say maybe two plausible scenarios here. So like the first scenario is this perception engine stays like basic at simple nouns and what's not but perfect. Um the current models us some three they're good enough but they're not nearly perfect. For example, on our benchmarks we are 68 in terms of F1. Some three is 62. But like for it like to to be like you know, I would say deployable, usable by bigger models, it has to be like more than 95. Um it like essentially we have to treat like this basic perception as we are treating let's say GSM 8K now in math. It used to be a hard benchmark a few years ago. Now each and every model scores like almost like almost 100% there. So this is the first scenario where these models like they do the primitives, simple nouns, they can track, they can do OCR. But the orchestrator does the high-level intelligence like the planning, deciding what to ask next, and does like the reasoning to manipulate the outputs. So this is the first scenario. The second scenario is the perception engine what we are having now it like expands a bit. Um we still like beyond like simple like you know, segmentation. Um but not do like does not expand to do math like benchmarks like MMU or Math Vista like geometry. But it learns like what you call like like grounded captioning, grounded VQA where it can answer um like in like in evidence-based manner like based on real measurements, based on like the segmentation mask or like all all the pointing or whatever. Um I would say it can be like a VLM but for special interiors. So this is like the two scenarios I see. Um in both scenarios like one thing I'm confident like in both is um that even perception like it has to be agentic on its own over images. Which means given an image it has to be able to do like you know, zooming, cropping, do some operations, verifying, and what's not because it has to be robust. The current like models we just like give the answer there is no thinking behind behind it. We just give the answer directly. Which means it's more like prone to to to to make to make some mistakes. So in my opinion this kind of division is not clear. For example, like the the demo maybe you have seen on um on on Twitter like with Gemma 4 and Falcon perception where like we we act in agentic way to essentially like the question is which bird is flying the highest? That's another yeah. For example, like any of these any of these demos. So like this is where this boundary comes up, right? Because essentially to solve this question like for now like we need to segment the bird compute like like simple like geometry proxies and then give the answer. But the real open question is do we actually need this many kind of explicit like back and forth calls between the orchestrator and the perception engine? Is this computation? Right? Or can a small fast perception engine actually reason to like can learn to reason over its it's like latent segmentation tokens to do this like internally like to compare and what's not to answer these simple questions. So the boundary uh might be like possible by like essentially how much ground reasoning we can like like fit in these models. So essentially what does this is essentially what we are doing now. And um we're exploring these questions. We'll see like up to what point we can we can push. And once we can't push anymore which means this is like the boundary between these models and and and the orchestrators. Cool. I I like it. It's very a very pragmatic uh answer here. Um this is pretty cool. Um what's what's your your take on this on this reality? My my take is uh completely aligned. And I think you you you you mentioned an earlier point about what if the these big frontier models they they they would just be what if they just be as good as like smaller model like this? And I think I mean they can but from the evidence we see at the moment the vision kind of meh and they are prone to hallucination. Right. We we we we we can have like argument hypothesis about maybe they are not because they are late fusion and maybe early fusion can have an advantage. Maybe they can just don't have as good data or maybe they just not focusing on these kind of task at the moment. So so maybe the opportunity is still there for for us to provide something let's say very good for the frontier model to even bother improving the model while they can just go on or something. So that's the kind of approach we are we are we are taking here. I think like I think there was a paper you shared yesterday about like the fact that um if you just remove like the the text information uh most of these like kind of frontier models just collapse and like they're they're not really like doing much. Maybe like the language biases that they have um is way too much. And when you look at the architecture that you guys like did for the early fusion stuff um it it is really different than like what the a big language model would do. Like there you can see like the the the two kind of um encoder-decoder type of stuff like a being put close together. But still it's fundamentally different than what the uh what they are doing. Um I think to be honest like this is like you know, we're going to say we like a bit like a bit of like speculation in here. So if like I think I think it all all comes down to the focus, right? In the sense that like let's say you're like you're OpenAI [clears throat] or you are like Anthropic. And you want to build models like there are two axes like there this is like the task like how how like um how economically important is the task? But the second axis which is very important is how like is this task actually um does it give you access to the final user or you have to be part of a system to serve the final user? For example, the tasks we are talking about in here like grounded reasoning and what's not this is let's say most of the applications, right? Let's say you want you are at a pharmacy. I'm going to just give an example like at a pharmacy and you want to do like you know, assistance of like you know, counting and what's not. So this forces these people like these companies to not be this the final system but work with other companies and other people to help them improve their their systems. Which is in my opinion not what these companies want after because they wanted to go so so fast. They wanted to like to grow faster to the point where they want for coding and this kind of like immediate impact from an economical perspective but also they have a lot of degrees of freedom because the system is controlled. Now for computer like for what we are talking about like you know, all like the segmentation and what's not um I think I think it's it's going to be there for robotics and what's not like or like OpenAI for example, they they published a paper recently called GDPV where they actually start like talking about like you know, going for economically like viable I would say use cases. And then the the question will will be there here. Because these models also they are agentic now. Will they try to build this kind of capabilities as part of the big model which might require some changes or to the architecture like you need to add segmentation. I'm not sure. I'm just like I'm just speculating in here. Or they're just going to go there and build a subagent which is like specific for perception that is fast, reliable, and it can accept changes to the architecture. I'm leaning towards the second option. So this is what we are trying to achieve in like mid mid term. So let's see. Yeah. Makes more a lot of sense. And a very pragmatic take again. I like how you guys thinking about this. Um it would be really cool if you guys could present um like [clears throat] the paper. I don't know if you had a slide prepared or um you have a noted paper. Um how do you want to go about like presenting them uh the architecture? I can't share here. I think I have like few slides in uh for the demo. I'm going to send you the link. And we can just do it all over the paper as well. Cool. That'll be nice. We have a question here. I think so. If you can share your screen. We have a question on Twitter. It says, "What's the limit of object you can track with Falcon perception?" I think they're is thinking about like the amount of dense mask you can have at one shot. Didn't you guys went to 600 at some point? Yeah. Okay, so that's it. You mean like you know how like the how many objects we can do we can do? Yeah. Yeah, yeah. Essentially like I mean this is this is like you know if you go to the differences between the architecture we have on some three detail I mean in general. Detail they have like you know fixed number of objects queries, right? Like for example some three has 200. Means the the best you can do per like per prediction is 200. But for us like since it's auto aggressive we can do like as many as our data data sets allow so. On our we are doing you know we can do like you know about Cool. I still need permission. to From me or from I guess I don't know for just a second. Okay, maybe I can share the paper first. Yes, so you want to talk? >> [snorts] >> Try Falcon perception. Okay, we are there. You want to talk Yasha or Go ahead. Go ahead. Go ahead. Okay, so maybe I just quickly go over the architecture. So here as we said we have only a single transformer where we linear where we classify the image flatten it and prepend it at the context followed by the the text and the task token. A little bit different from normal LM is we for the prefix image we let the attention mask to attend jointly bi-directional for for for both uh uh for for all the image part and then this will be followed the following token will still have a causal mask uh like a normal LM. And this kind of approach is I think first proposed by by the Gemma. Uh they call it the prefix LM mask attention and you Yeah. So So first we have an image and then we have a few text prompt. For example detect cat. And then the model will condition on the image and the prompt to predict the next token. Here we will for for each instant we will always predict three token. Uh which we call the perception the chain of perception. First is the core token. Then the and then the side and then the final segmentation. So the core token corre will be decoded into uh two value X and Y which corresponding to the center of the instant. And then the actual numerical value will be re-encoded into the hidden state vector bypassing the normal token embedding. Uh at the the start of the next uh uh at the start of the transformer and it uh will predict the next token corresponding to the side but we will actually decode the final hidden representation to the height and width. Uh so together we we will have the the center and the height and width to to localize a bounding box. And then the the model will predict the final token for the chain of perception. Uh we we still take from the token embedding but it will condition on the previous uh image prompt and the localization uh of the bounding box here and and finally we will take this uh segmentation token to decode into the fine uh the fine grain high resolution uh mask. And to be able to do that we we know one thing for sure we need is a good image feature. So as you can see here after the transformer we take out the the token corresponding to the image. We pass it to frozen uh image feature up sampler. Here we use the any up to uh to uh to to to get a high resolution image feature. Here is a PCA visualization for example of of the input image and you see we have a very good PCA visualization. So that it mean like the the image feature after the transformer still contain very high and fine grain uh still have a very uh the high spatial coherent in it. So this uh spatial coherent image feature allows to easily create this high uh this high resolution feature map by simply adopt product between the segmentation token and this uh image feature map. And for different mask we would decode to different chain of perception core side set. And we will use different uh segmentation vector dot with the same high resolution image feature to produce the corresponding instant mask. So so here like uh the the the the the the main point is that the LM will handle the localization and the feature of the segmentation token. And everything is done in the LM framework auto regressive so everything is conditioned on each other. So the side of the B box is conditioned on the where is the center of the B box. And the mask is conditioned on the on the actual B box. And like the instant for example of of a different cat is conditioned on like which other cat has been predicted before. And uh so one single transformer >> True. True. Yeah. We will we will we will handle all the on the let's say the on the difficult aspect and we have different lightweight head to uh let's say like inject extra inductive bias into our model. For for example for for this reason we can have an LM that can do uh segmentation mask in high resolution but very quickly. Instead of having to predict 100 or 1,000 of polygon token for example. Mhm. So that's essentially the idea behind our design of the architecture. Let one transformer handle both the complex part about understanding image query and conditional prediction. And just have lightweight head to inject the the extra inductive bias we want so the model to have about like that uh understanding the the Yeah. It be able to understand and code the the location effectively or be able to decode high quality mask efficiently. Yeah. And and from this structure like it's very clear that like uh the like a image I kind of have embedding are just computed once and then like you get like the the up sampled uh element that you need to like do the masking, right? Um Yeah. But then you have your chain of like a chain of perception. Um From the paper I've seen that you use like the raster ordering to decide like which mask you're going to go first, right? And I've seen the plot where like there's better performance with this. Um but do you see a degradation in a performance on like the uh like this I think it's top-to-bottom, left to right like on the the one like uh at the end completely. Like once like he has his condition on everything, right? Like not that on the average, but like it's just the last one, right? Because you're you're you're far further off and the in the context. Have you run some experi- experiment here on like um on these individualized uh mask um errors? Uh You want to talk on that, yes, sir? So, uh the question was like why like how did you end up doing raster, right? Well, I I I know why you're doing raster because it's like it's better, right? Than uh random, right? Of course. And like the size. But what I'm asking is like because in this specific scenario when you're using raster, like I think it's like the bottom um uh bottom right will be always last one that will be detected, right? So, if this is the case, like do you see is there any potential degradation of like these ones um compared like to the fresh uh top left uh that you have to uh detect because they're they'll like basically at this point in time the model just have to do like a simple task and it's conditioned like on the on like like the image and like some some some of the stuff that you put there. But as you go further on, you have much more uh that it is conditioning upon. Um is it is it is something like this happening or is it actually like I don't know the opposite where it's actually better? So, yeah, for for first like you know, why why why we like why we ended up in raster because they're related actually. So, when we started this project like when we started doing like the trainings and like setting up the uh setting up the parameters and the choices, which is myself and Phuoc, we were convinced that we never do raster order. Like >> [laughter] >> because like it's Here we are. It's like it's kind of like it is not not not not not desirable. And the reason is cuz we wanted to go for like random because if you do random, you kind of enable kind of um like kind of in-context refinement. So, the first object is kind of refined on the second object and what's not. Um this is like that was like until 80% of the uh of the trainings. We were like always training like with the with the with random and what's not. And then one of the days uh actually Phuoc was doing the analysis of like you know, you take the first core token and you see like the probabilities of the core head over the next ones. And we see that actually the model is not doing a lot of refinement. The model is too confident that I'm going to do this object, I'm going to go to the next object and what's not. There is no like self-ref- It's not probabilistic. It's not like you know, 50% this or like 70% the other. And then we say, "Okay, so we kind of know why this is maybe some some mistake we have done in the architecture like the way we like kind of factorize the bins in the core head, but it was too late because it was designed and and implemented and what's not. And that's why we went for raster. So, we went for raster because of our design of the core head. And because the model actually was not doing a lot of refinement. And so, like the quality of like top left is pretty much same as bottom right. Uh so, yeah. Yeah, I I want to expand on this a little bit about the difference between side and raster because our particular choice of uh decomposing the an instant prediction into core, which is the the center, and then the side, and then the the seg, which is the final mask. So, like uh for example, if we we do the ordering as size, when a model look at the image, it the first instant it has to predict need to be the the instant with like the Yeah. Has a had a larger size. But the side token is only predicted after the core token is predicted. While if we do raster, uh the ordering of the the the raster order can be determined just by the the the core token, the center of the uh the the the the instant. So, but because we we we we we make the architecture always predict the center first, and the raster order align with the the center. So, the model can learn the raster order just by by the first instant. Uh by by the first token of the of the three tokens. So, it I think that's our hypothesis of why for our particular model this time the raster order were bringing such a huge improvement. Yeah, that makes a lot of sense. Um I had also another question about the uh just like the general idea of the architecture. If you can go back to the main image. Like um uh it really looked like you took like the kind of encoder-decoder kind of separated architecture, right? But then you just like jammed it into like uh like the the same kind of model. Like what was the rational to um I know that like you you guys look at Cali Jiman and like he he kind of it was working there. But what was the rational for like having it um like this in in um in in this particular configuration with like um the the the head helping in this flow? Like um how did you how did you go from like like literally nothing to uh this figure? Um maybe talk about the our our head. For example, like the the the center and the side encoder-decoder we have to give uh credit for MoonDream uh because uh they they we we we we studied this specialized head. We we were inspired by MoonDream, too. Even though so, I think they have they predict two core tokens for X and Y and then one side token for H and W. Uh in our case, we were inspired by it, but we just like uh cleaned it up a little bit and we decided just one core for X Y, one side for H and W. Uh and we uh uh we we we we use the Fourier feature as for uh to design the the the encoder so that the model can have the best chance to to to to to to look at the actual to to encode the actual numerical value. And uh And then and for the segmentation head because we we really don't like the idea of predicting a huge amount of token just to form a polygon. So, we know we need uh uh uh an a design that need that can generate all the mask at at once. And for that, we we know we would need very good image feature and high resolution as well. So, high resolution will impact the the the final detail, but we know we need an image feature uh that's like has very good spatial coherent and spatial feature. So, that's why we uh we we equip the our in the attention layer the the full mask over the the image token because uh so that the the representation between uh each token can attend and uh and then uh refine each other. And we can do this because for image token in during training, we do not put a loss on it, so there is no no break of causality as uh basically. So, yeah. So, that's what the the the idea behind the architecture. And we tried also actually like maybe for for like maybe a month and a half or even 2 months, we tried actually being causal because we know like doing like you know, full [snorts] followed by by causal is not uh I would say not integration friendly because you have issues with now with VR LM and llama.cpp and all this kind of frameworks. Mhm. So, we tried for a while like doing causal like um like NV2 from like there's a paper from Apple called NV2 in which they do a kind of next patch prediction. So, each patch predicts the next one. And we had like this loss where we were doing like you know, like next patch prediction and what's not. But because we do segmentation, we had kind of kind of leakage in the attention. So, it was learning, but not that good. It was just okay. And then we went like the causal prefix LM where we randomly sample like a value between 0 and 1 and then we do like fully causal uh like full mask up until then and then we we continue with causal and we apply the image loss only over the remaining part um of the mask. And the reason we do this because we wanted to be able to switch in the mask any way we want like in like either direction. We can go like because we we we we do hybrid over the same image like at at pre-training which means at sorry at inference you can either go fully like block for image or you can go fully go We tried this kind of things but it was adding a lot of complexity to training and it was like making the training unstable. So at the end of the day um like because we start from the distilled model like for like this models they are like actually using like you know full mask so we had to adopt we have to we had to you know stick to this kind of mask for for images. So I think now we are trying to for this current model like go causal and fine-tune because ideally it should be causal actually. Uh because it's like integration friendly but let's see where where where we go from from here. Wait wait wait. Um that makes a whole lot of sense. Um I have a question about the chain of perception right because you have the coordinate coordinate size and then like segmentation. Um I think you have tried you tried just the segmentation right but it was way too heavy. Can you talk us about about that because I think like the the actual funnel is just like to make the the problem the segmentation kind of token uh that it has to do be much simpler so that like you can you can simplify a bit the architecture for this. Is this was this the reasoning that you you were going for here? Um essentially like like at the beginning we did not have this like chain of perception. We had what you call like the object token one one token. We wanted like you know every instance in the image to have like one specific token called the object token and do the dot product and what's not. And earlier on in the training when we started with this idea um what happened is like because you don't have any kind of encoding of this like you know the masks the instance masks like what happens is when you train the model starts predicting like semantic masks instead of like you know instance masks for the entire so person person person does what like for the first person and the last person they all share share the same semantic mask which is like you know over all the people in the in the image. So when this happened so and then we we have two options here. First option is we got like a masking coder. The second option is we condition based on coordinate size based on the B box. Uh the reason we went for the second option is obviously it's lighter because you have like just like two values instead of like you know encoding the entire mask but also we wanted to give like once we train this model we can easily turn it turn off the segmentation and have it as a detection model which is much faster of course. So we from the like you know core size core size seg and then simply do not decode the seg token keep coordinate size and then become the detection model a faster detection model. So this is how we ended up like with the with the chain of perception. That is pretty cool. So and also just generally like because I there's also like a whole bunch of decision in there like there's like the the up sampler that you decide to put over here there's like the native resolution right for the for the stuff there's the 3D rope that you decide. How did you go about like from from like the basis that you had it to start to select which which element to to try and add in into the mix? Uh to be honest a [laughter] lot of the choice here. Uh like vibe based. >> [laughter] >> It's always like this every time I ask this question people are saying like yeah. We we thought this one was like looked better into the the system. I love it. Yes. >> a saying in the in the team we we say you know you start with the most um correct place and debug from there. So essentially like like yeah it's like I would say I would say most of the so we had we had like you know when we were starting doing this we we wanted to be kind of following what's happening because it's actually so much aligned with the with RNNs. So we follow the current literature we we follow the best practices and obviously like you know reason about like our our our targets what do we want in the architecture and use it there. So for example like the 3D rope and what's not this actually was part of the distillation project in which it actually proved to be helpful because it gives us gave us the chance to essentially for distillation maybe I'm not sure maybe going to talk about it later but for distillation like you take people like radio like uh some some other works they do it like stage wise right? They do like start with the small resolution 2 2 256 they go to 384 and they keep like increasing the resolution. That was not our the way we have done distillation we have done it like you know with mixing all the resolutions from scratch because we believe a pure I would say training recipe always wins. But if you do it like this which means you're going to have no issues with like with with with the with like interpolation of like the image like the image representation from small to big and what's not. So in distillation when we tried like 3D rope and this kind of this kind of this kind of techniques what ended up happening is we train let's say at 1024 marks resolution and then at test time we we we we blow up the image resolutions to 2K or 3K and we see that the PCAs they get better with with higher resolution. Right. Which is something you wouldn't get with like with 1D rope. So that was the the reason behind um behind 3D rope so essentially like it was like you know one of the most I would kind of set of the art way of doing like positioning codings if you want to do like images like mix up images and at native resolution. But so this kind of I would say this kind of like reasoning behind how do we choose things in the architecture it has also downside because sometimes we try things uh for example our activation we went from Swish to ReLU squared. Um the reason we went for ReLU squared because um like in the nano nano GP like in in Andrej Karpathy code base they people like praise it a lot. So we trained with it because because of the premise it's it's kind of faster and what's not. But we ended up having like you know like large activations in the model which made it hard for quantization and what's not. So yeah so I would say the lesson here is um like it's good to always try to go for the latest techniques try to implement them and improve them and what's not but we have to be very careful when it comes to kind of very well established I would say techniques that are already there and changing them that will have like that might have some side effect I would say. Yeah and there's so many moving part also like in this thing like if you switch one thing then maybe something else start to like not function properly. Have you have you had to do any gymnastics with Me One whatsoever to like make it work because like with you had to switch I think you switched from AdamW to Me One and it was kind of better and more stable but I know that like every time folks need to like especially in language like they they they use it they have to do a whole bunch of work uh to kind of like stabilize right clip it or or whatever but then they get very smooth kind of loss curve. Did you have to do anything to make it work here or Yes so for for for Me One we we we we tried a lot of thing a lot of different implementation uh but we end up using the the Deion package from Microsoft. Uh that were like but we still use the Me One optimizer from that Deion package. And they were a lot of blowing up more than not converging loss blow up etc. but then we settle on the Kim Moon light uh scaling uh rule for for Me One which very conveniently allow you to reuse the same learning rate of AdamW. So the moment we switch to that scaling rule of uh of Moon dream reuse the same learning rate everything smooth. We immediately see like a big improvement over over the loss. I love this. >> [laughter] >> So um for us like to be honest like the usage of Me One was was almost surprising in terms of performance because because like we have these specialized heads and we have like multiple loss functions like for segmentation and for next token prediction. Essentially like the performance was like day and night. It was not it was not was not even close between Me One and AdamW so it was kind of surprising to be honest. Yeah I saw the graph like the the gap is is is really there. In the paper it looked like you just plug it in and it worked but it seems like it was kind of a bit the case for this. I have a whole bunch of question about like distillation but maybe you can continue I think you had a slide that you you wanted to show. So if you want to go through this and then I can ask my question. We also have a bunch of questions from the community, but yeah, we can go through the slide first. Uh, okay. Do you see the the slide or do you still see the image the the paper? We still see the paper, yeah. Okay, share this tab instead. Okay. Okay, perfect. We see the slides, yeah. So, the slides I just wanted to put a few examples in which um So, maybe maybe maybe before we jump into the slides, maybe you should like define P bench if uh if if you want to >> idea. Yeah. Yeah. So, essentially like if you go back to the paper um Look, yeah, for P bench. So, when we started like doing this um like obviously there are like simple nouns, right? Like you know, you can just you can stop there the uh table. Yeah, the table. Essentially like uh if you are doing like you know open vocab segmentation or referring expression segmentation, so essentially you're asking a question you're asking to identify either like a noun, a concept in the image. And for this like users like have many degrees of freedom. Like they can ask like varying stuff. They can go from simple like nouns like car, phone, whatever, as well as more I would say compositional uh compositional I would say concepts. So, we wanted to before before we start like training the model and even like benchmarking and seeing the current performance of the models like back in the day before some three I'm talking Moon Dream and Quan and and and Gemini. We wanted to essentially understand from a fine-grained perspective where these models work and where they don't when it comes to this like you know task. So, so starting from simple words like you know car phone laptop. So, this is like what you call simple nouns. And then users can actually add attributes this nouns and give an adjective. Like let's say a red car. Um, like a blue phone. Okay, the color can be an adjective size the larger the larger phone, smaller one. Um, it could be like the adjective or the attribute like can be anything like user simply um can identify whatever they want and ask the model. And our models have to be robust into like answering such such stuff. It's you this kind of stuff. Uh, like a door open, door closed like like the the the article the functional behind the behind behind the query. But also we then we realized that actually one of the attributes which is very which people actually ask a lot is using OCR as an identifier. You have like shelf or in a supermarket and people ask uh Coke or or or Pepsi. So, they're not interested in all the cans, they're interested only in the Pepsi ones. So, the model has to identify cans to read which ones are are Pepsi or Coke and just segment those. And then we have level three which is when people start like giving the sense of direction to the model like left, right. Like you know, the model has to be like specially aware of what's going on. And the fourth level we were interested in is what you call like relationship actions. When a human being is actually doing something which means it involves like you know relation between two kind of entities in the image and one of them is operating using the other. Like where the model like this is not a video. We are talking about images now. The model like just like has to understand the concept understand the concept see like based on the context whether which person is talking among the people. So, these are the four levels. There are other levels that we identified require like a bit more reasoning. For example, level five would be you take there's like some dirt on the on the floor you take a picture and say segment like the area to be cleaned. So, the model has to understand that this is the area, this is the dirt, and this is like what what should be cleaned. But that was not like actually done for the for the sake of this this product. And then we we start doing like evaluating models and see like where models work and where models do not work. And we see like a clear like a scaling between level zero to level four where level zero is you have like certain performance and starts going down like among the levels because complexity goes up. So, essentially like this is how we built our our benchmark and this is how we built our capabilities. That's what this model does. And just as a side note to our surprise actually by the way that just by doing this we realized that most models including ours for now they struggle a lot with level two which is reading in the wild. Like we see like you know huge drops like for most models for some reason. Um, when when I started when we started doing this we were thinking oh, like identifying the sense of like left, right, bottom this kind of stuff might be harder than than just reading text, but it turns out the opposite. No. Actually reading text in in in in images is like much harder as compared to identifying the sense of orientation because text is like it's high like high frequency like noisy data. It's to have multiple things like M and whatever. And uh this drove our data generation uh annotation and what's not. So, maybe if you can show the slides that I'm going to But I like this part is very interesting that level two is hard because when I was reading the paper like I know that you guys like did a very smart like distill of Dino V3 and then um SigLip two in order to get like SigLiNo and then like this was the weight initialization for the model. Um but then you didn't do that for the Falcon OCR. Right? Do you think that this has to do with it like in some shape or form that like this this very high bias at the start um toward like these more I don't know like natural kind of shape versus like the language thing which is like much different. Um, has something to do like a I don't know like did you try to just randomly initializing it and see if actually level two would be better at the expense of like maybe the other levels? Yeah. So, like the the actually the reason we have done distillation is precisely to improve level two. Because our assumption was if you have SigLip which is trained on captioning trained a lot of data so it should actually start learning to read things in the wild because this is how captions are. But as you mentioned OCR was trained from scratch. But this So, like it turns out actually like reading in documents like where the background is kind of like you don't have the noise coming out of from the background or even reading in the real world but like a blobs of text you have in the real world but you have like some part of the text written and you read it is completely different task as compared to reading on objects where you have like the background, you have the color of the object and what's not. So, it turns out this specific part of OCR is the hardest one as compared to the other part of OCR And level two does not do like text like a blobs of text and what's not. It uses OCR as an identifier on top of objects. Right. So, like um So, level two is still even hard for our Falcon OCR model. Like reading in the wild. So, it should be level five then. >> [laughter] >> Why you put it level two? Put it level five. Cuz it's an attribute. It's it's OCR as an attribute. Yeah. So, that was that was like we are surprised like that was surprising actually for us for all of us in the team. We were thinking oh, level two is easy. Let's let's worry about level three and four. And then nope. And these model, huh? Um, and this is pretty cool. And before we move on like can you just explain like how you guys like made it uh PB bench. I know that you have another paper on that, but I really liked how like you were able to I mean like in century where you you leverage other models to kind of go and like like bootstrap like the the the the segmentation and stuff and like the the the labeling. Um, but then you didn't discard those that the model will not were not able to do. So, you still have very hard task that are rooted to toward humans. I really like this. Like can you explain it a bit how this whole uh benchmark was was created? Um, so the benchmark was So, this is what you just mentioned Justin is for the training data. So, for the benchmark was um so, we have we have like a bunch of images and it was like create annotated and created by the team members from scratch. So, we get an image and then we say we want this image to be in level zero or level one or whatever and the people like they look at it and they create the query and do the segmentation. By the team member because early on when we are doing this um like looking now it looks like even [snorts] for us it looks like kind of clear, but back in the day like six months ago it was not clear what this benchmark what this levels are. So, we did it like the benchmark was done like inside the team. And then moving and then we try to replicate the the way the benchmark is I mean the level in the into the training data which means when you get the image we create like you know, level zero, level one, and whatever. And we know we know for example that like um like uh model like some some was not that our V2 uh Moon Dream is very good at level zero and one. But we know also that Moon Dream struggles with level two. Uh so which means Moon Dream will be discarded when we annotate for level two. Um so this gives us kind of profiling that served as as as a signal to like um to like direct the data annotation. But coming back to your question, um so for data annotation, what we did is we did this kind of consensus between models. So we take we take like a lot of data and we like run like uh detection detection because these models are for detection like to be clear. And we do like you know, we do like measure where where they do agree upon image and query. All right? But obviously we cannot train on this data because this is kind of the the easy-ish data. So our model will never be better than these the models used. But for what they don't what they don't agree, there are multiple categories. Right? So they cannot they cannot agree because the IOU is is is lower than the threshold which means model one has a certain box, model two has box on top of it, but it's a little bit larger. Which is one one one one option. They can agree on one object but not the second one. This is another option this is another option. But this is not what we did for human annotation. For human annotation, we went for cases where um it's kind of binary. The two models have literally zero intersection. Right. >> Model A is saying this object or this query is here, model B is saying it's there. Which means one of them is completely wrong or both of them are completely wrong. And by doing so, we are sure that we are kind of patchifying in the knowledge where these models do not work. Which means if you train on such data like by definition our like our model has to be has to be better. So this is essentially how how it was done. And this was done per level. Okay, that that that makes a lot of sense. Um uh How how how come like this this model that you have for Falcon Perception is so good at like the uh area where like the other are struggling um because literally like they they were able to see like these hard tasks also. Um This is fantastic. So guys, you can continue. Sorry for the all the questions. Uh Yeah, great question though. Okay. Uh Maybe you you actually you want to talk about this? Yeah, so I just want to slide. Yeah, I just wanted to show what do we mean actually by levels. And the reason why we show this is because for people listening um just to give them an idea of how to use uh Falcon Perception. Like where what it does and like I'm just going to stop. So I did not do level zero because I think level zero is like this is this is common for everyone like just like just like common nouns like uh this is obvious. So the image you see in here in the prompt is what we call level one. So you have you have this kind of um this kind of candies. And the user can ask like we're not interested in all the candies but we're interested in the white uh candy coated chocolates. So this was given to our model and you can see that our model only selects the white ones. Like uh over the marks. Right? Not not other colors and obviously like people can ask for other colors. So this is level one. We are using attributes to retrieve the objects of interest. Uh same thing from the second image. Um which is you know, you have like you know, a few birds in there. Um the user can ask seagull with the yellow beak. So among all of them, which is the one with the yellow beak. So just to illustrate to people, this is exactly what do we mean by uh [snorts] what do we mean by by by level one. All right? Um the slide three is is is level two. Okay? So for example, you have this image of I suppose some some shop. And the user can say Tokyo 3 hoodie. So the model has to look look for all the hoodies like like and just to retrieve the one where Tokyo 3 is is written. Same thing for the next image. Um Like like this is like you know, um like you have a race and the person can ask the Jaguar Bud Light race car. So this is Jaguar on top of it, Bud Light. So the model has to read to retrieve. This is level two. Obviously um these are like I wouldn't say cherry-picked examples but I should have to try four four examples to select two in here. So we are still struggling with level two. Such that like we're like even on the benchmark we are better than some three and other models. But this is far from being solved problem. Right. Um and we are targeting this for um for for for the next for the next iterations. All the other levels, it was like literally first shot. I did not even um like uh uh cherry-pick. So it was like the first example it passed in the model. Yeah, so there really something about this specific task that is a bit different than others. Um Uh just like random uh idea completely. Would it help if the model knew like ahead of time what type of level it has and like inject that biases uh straight into the um into the model because I would assume that like it just from the prompt you could kind of figure out like uh in which kind of cases. So here the Jaguar Bud Light race car. Like you would you would be able to say like yeah we're on level two because I will have to read stuff, right? Um do you think that this will will will help in any way uh the model to kind of like improve without changing too much in the in the architecture? Yeah, definitely. Yeah, definitely. I think yeah. Yeah, I think I think yeah. If you give like Bud Light maybe maybe we should also ask um like you need COT there, right? Because the model has to read like few other stuff and select uh even if you do it like this. Yeah. Yeah. Yeah, we'll try. We'll try. We are we are trying to solve this in different ways. >> [laughter] >> Let's see let's see let's see how where where do we reach. Like it's it's surprising like even Moon Dream and other models they struggle with this. Something to it. I don't know why. It's something yeah. So then like level three is the next example. So this is like you have a building and the question is like potted plant second from the right. So you have like go to the right and the model has to like this is the first this is the second. So that's the correct one. So this is level three. Like you know, like second person like left right this kind of stuff. Um same thing for the >> like it's absolutely crazy that this is easier than the first one. The the the the one just before. It's crazy that like it's able to look at this and like potted plant second from right and then go here right away. Um I think this text is high frequency in in the wild because the background is literally acting as as noise and the model is not able to essentially read for some reason. Maybe resolution. Maybe we should go to like even higher resolution. I don't know. Uh we'll see. We'll see like how do we solve it. Um but yeah. But this is essentially what that was the goal behind P bench because now we know where the model works and where it doesn't. I mean like gives us a signal. >> [snorts] >> Same thing for slide six. So that was the image and we say white car in between uh two black cars. So there is another white car but behind it there is a bus and in front there is a red car. So the model knows that this is the car we are talking about. Um so yeah. So this is level three sense of direction. It has to identify stuff. In level four is there's a person playing trumpet. So if there are four people, it identify this one correctly. Um this is level four which means there are the person is actually doing an activity. So the model has to identify the activity and like identify correct like um like person doing whatever. Um So this is level four like person walking as well uh on the sidewalk. It missed one on the the very far but like it's fine it understands like this is the sidewalk. We're not talking about the two people like you know um riding the uh to the horse or I think you got it now. Like I see the mask on the on the very far. Um I think yeah. Okay, now this one. missed one Is he walking? I don't I'm not sure if he's walking actually. He's waiting for the bus. Yeah, maybe not yeah. >> [laughter] >> You you would be an excellent labeler of this kind of data. Yeah. If you need help guys, let let me know. I'll be happy to to annotate a bunch of stuff. So like if you can go to level slide 10. So this is what uh one thing that um why we why we do like auto aggressive, right? And why we do auto aggressive while having the chain of perception. Because auto aggressive obviously gives us a chance to do like as many objects as as as our model allows or dataset allows, we're are like bounded by let's say 200 object queries. And then you can do it in a single shot. We're not also some three for example you can do it, right? But you have to tile the image. Like you have to tile the image and then you have to if you tile you have to resolve all the um all the issues that come at the at the at the borders like over like overlap and what's not. But for us what was it surprising is um like the model actually generalizes to do like you know 200 300 up up like for now we are even able to reach like 600 um I would say uh 600 predictions without tiling without like doing any like just one shot. It goes like you know auto aggressive it does them all. You see this is like the coin in here like it does them like most of them I don't know how many are in here but like a lot maybe more than 200. Same thing for the birds. Um the example like in slide 11 um like same. Like yeah. So this is like the lemon I suppose. Yeah. On the tree which is like harder obviously like like the like it's crowded and what's not but the model still does it. Um same for like the image on top. The next image as well was um was really surprising like you know I think there are like almost 300 300 uh things in here. And it's single shot like auto aggressive shot and the model is not repeating. Like it's not like you know it's like goes into this repetition mode where it's like you know collapses on the single thing. Um which was really surprising to us. So this shows us that you know uh we don't know if early fusion really helps here maybe it does because you know the model has access to all the features but at least what's for sure is chain of perception makes it like efficient because essentially if the model produces 250 tokens that uh sorry 250 objects for us that's only 750 tokens. Course I said um that's it. 750 is very fast to generate as compared to let's you do like polygons and you have to generate like I don't know like thousands thousands of tokens to do this image. Yeah. So and obviously auto aggressive because we don't have like essentially like you know a bias over how many objects we should uh we can we can produce. So um yeah. So I just wanted to show these examples um which is which reflect the reflect the best mark. Um this is essentially what we expect um people to use the model for. Uh just a caveat you said in the slide but I think you actually have mentioned this is from our uh reinforcement learning improvement of our current model which has not been released yet. Currently the model is still struggle a little bit can sometimes collapse on these dense case. But our early RL experiment already showed that currently in there we can Exactly because of the pass at K thing, right? Like you did pass at eight and then it was able to kind of like find these but it need like a few try. So then like with the RL you able to kind of like get the pass at K in pass at one and then get it there. Okay this is so cool man. So uh cool. Uh um uh Do you think it will help like if you had like I don't know I know like adding tokens for the stuff is um is dangerous but in my view for like level two it seems to make sense to have like hey where is the object? It's like here and then you have the bounding box and you have I don't know like a captioning token or something like that and then you have the segmentation so that it knows kind of like the language can be injected in here to kind of modify it. And if it knows early on like if the model is able to know am I in level two or not, right? It means that they can kind disable this one if it's not it's not needed and then you can get like just a normal segmentation behavior um without having like this kind of middle token in the middle. But it seems like uh the uh the the problem is maybe too hard when you look at like the whole thing and you have like all of the the background affecting but if you're kind of like a limiting yourself to to like um where the bounding box is uh maybe it will be a easier uh easier task for the the model to take. Um Yeah. I mean look this is what we are doing for V2 actually. Like uh the second version it is like as as I mentioned in the beginning we wanted to be in take we wanted to crop like detect things before before it does because we want this kind of reliability. We don't want to essentially ask a question like for level two three and four and hope for the best. Like we don't know what the model is doing and like like we don't want it to be up to up and up to our our training data. Because if like we know it's kind of very hard to do it like for SFT for sharing so we knew like reinforcement learning because it generalizes better supposedly. Yeah. So uh yeah. We are doing this for V2 as well. Yeah. That was that was a I mean like we can go into like the few question I had still. Um the reinforcement learning learning part is is kind of very interesting for me and like the fact that you're able to get these results with like that stuff. Um uh really make me think about like um uh R1 because you kind of in the same shape before reasoning kind of get like arrive and then you can have like these kind of composition of action in whatever kind of token space that you define, right? In language like we're just going to talk and then like I'm going to make a plan and then like it will scaffold myself. It will take more time but it will be higher quality. But it seems like you have all of the the ingredient to be able to do that and have like the the token being like generated in order to have a a better uh better output. Um so from the earlier experiment here like how how does the RL work here? Like what type of um of token is it outputting in order to get like the uh the thing back and like what's the what's the reward signal here? Yeah so like the current reinforcement is not like really reasoning what we are doing. It's mostly inspired by there is a work from Lucas Beyer um about like you know uh like using reinforcement learning um What's the name of the paper? Uh uh training computer vision with with task rewards. So because the way we train we train with with cross entropy and what's not but the way we are measuring performance is using F1. Which is might not actually align with the with the task risk. So for now what we are doing is we have like the you mentioned the case with like pass at eight and what's not. So we have few rollouts. And we want to make the right ones like more certain. So the algorithm is obviously like GRPO. And the reward function we are using for now is is just like F1. And like just like to balance between like um recall and precision. And we have like kind of we already have like few findings the model is much better at um at like long contexts and what's not. But for us like because of the architecture uh the obvious question is where to sample. Do we sample in the LM head? Do we sample in the chord head or in the segmentation? So obviously like throughout our trainings we realized that there is one loss which is the coordinate loss which is actually very indicative of performance. Means what? Means our backbone is the one doing actually most of the work. Like like the first point in chord like this object is there there here and the model does not struggle to predict size after chord and the model actually does not struggle to predict which segmentation token once the B box is correct. Becomes more of a refinement inside a B box. And for this what we did for now is freeze the the size and segmentation and the LM head and sample only in the chord. This is where we sample. Just like one one one one one like one place where we sample and the reward being F1 and we see like you know huge gains in just by this. So we see like you know the model becomes much more certain now over predictions. We don't uh there is there is one metric that we have called MRR. Internally it's not actually we did not put it in the paper maybe we should add it. And MRR simply means mask redundancy rate. You take your dataset you generate and you see how many how many and you do NMS and you see how many masks were like were were were were deleted. So we do this for dense for example the dense split. We see for example like models like a moon dream is around like 40% MRR. They repeat a lot when it goes to dense because it's harder now like you have a lot of samples. Same thing for Quen. Our current model is around like 10% of redundancy over like these samples the ones you see there like very dense ones. And surprisingly speaking when we do like after like GRPO the MRR goes almost to zero. The model stops like repeating at all. >> Wow. And uh which is surprising. Uh yeah. That's kind of crazy. Um Yeah um I had a Uh, I had two questions actually, like, um uh, if the coordinate uh element is the hardest, right? Would it help to you think that to give it multiple try in order to kind of like get it right? Instead of just like one shotting it, especially if like there's many of them, right? And then just like letting it scan uh the thing um multiple time while still being preconditioned by the the try that they had in the past. Um do you think that this uh this could help in the in any way or like we will confuse it more? Because now it has to keep track of like the first shot that he had and like try to correct it but later on. Um what's your take on that? Um I'm literally [ __ ] shooting you all of my ideas. >> [laughter] >> Yeah. See, I mean like for it like to have multiple tries, we need to be confident that the core token is being refined. That is in context. Right? Because when it's refined then like few refinement you can get it correct. So, which is not the case for the current model. The current model there is some refinement but not much. And uh we will fix this in V2. Like which is like we know what the issue the issue is mostly with the with the core with the core encoder decoder and the way we are factorizing. Um but yeah. So, essentially after like after we we include we have this behavior where it's random, we're not doing raster anymore and it's being refined. Which means there's a lot of in context, the model learns from the context like where where to look first and what's not and then trials and and all this stuff like make sense makes makes sense at this stage I suppose. Maybe for if you have some Yeah, because then like you you have like this reasoning shape because you could like put it into like yeah, we're just we're just trying stuff and then you can throw it away and and stuff. It will take longer but then if it is able to do this kind of refinement um then like uh my my guess is that whatever is working with RL will work even better and then it might be learning to refine on its own and then like uh then it's it's off to the a similar type of scaling and and shape that what we have right now. Um But just a better take on that. All right, I think this is a great idea. I um Yes, but like you in order to do that we must have the it I think it's belong to the general trajectory that we are trying to do which is trying to make the model a bit more agentic. But like I think your this idea is like fit perfectly into this framework because the model can predict core size sec for example and if uh it predict a little bit off the prediction is something very very jaggedy very high entropy and the model can have this uh segmentation token inside and then it will see okay this core inside during this very bad looking segmentation. So, internally it already know it is wrong as soon as it sample. And then it can condition on that, try again. Now it's wrong it try again but I think I think it it this is a great idea. Yeah, it it can work but it the model need a bit more capability. It need the agentic capability. Yes. Yeah. But like that like if I look at like everything that you guys have um it seems like all of the raw material for like that stuff to happen is like almost there. Um yeah, it's a very very interesting. Um maybe we can uh chat about uh a bit about the distillation uh process uh because I've I thought that that was um uh very very interesting and I I know it it comes from the MOEMOE paper that like got like uh then you define that as Signe now. Um but um uh uh I I I thought it was a very smart way of like getting the models started, right? Um and getting that that kind of knowledge in of these of these dense features. Um can you talk to us a bit about like that trajectory and like uh why did you decide to actually jam it in there for the stage one? Yeah, so first thing first, as you might as you as you noticed uh yesterday um the model I mean the paper title was AMWE and then it changed to Signe Signe now. And the reason we change is because we added like dense variants. Which is I can tell you a side story here which is can can I be funny? Um so when we started this project like during summer um we were not really confident about like early fusion that it's going to work. Um like just mixing image and text tokens and what's not. So, we spent a lot of time. Um this model up until maybe 75% of the duration of the project was an AMWE. Uh talking about both distillation and the perception model. And then we have like a perception model like trained actually up until uh up until convergence as an AMWE. And the reason we were doing this was uh we wanted to do like you know AMWE because just makes sense. Because if you're doing early fusion, you want like experts and then each modality can like take a set of experts. You can even like you know hard route like you know uh image and text tokens um like through all the experts which means you go from depth like image and then text to like width. You just do them in in width. Right? So, so distillation in the beginning was done in as an AMWE. That was the first reason. The second reason we wanted to do an AMWE because we wanted to do like multi-teacher distillation. And in multi-teacher we we wanted like you know kind of two models which are not really having kind of the same semantics. So, you want like DinoV3 which is very good at like um like what you call like local features which are great for segmentation. But you do also want image representations which are aligned with the with text concepts. Which means you need a model which is trained on captioning or like contrasted. Which is the case of um which is the case of of Signe. And we wanted like kind of this like image-text alignment plus like local features which means we we were forced to have like two teachers. And our assumption was like you do an AMWE because each teacher now can like you know can occupy a certain number of experts and then we can have like both um like I would say uh like both of them together. We do this training with all the tricks that you can see there in the paper. Um like we train like you know natively at like native resolution. We mix we will mix all the stuff. We like all the all the stuff we we have we have done in the paper. So, they were that was actually the majority of the work in the in the project and we spent a lot of time um making sure this AMWE is stable because you know training AMWEs especially at small scale is not easy. Like like load balancing all these issues. We had like all kind of issues with these AMWEs. And then before CVPR like um reviews came out I asked like it was an intern Sophia I'm doing this. I asked him like look we should prepare like few small dense models but you also should prepare like an ultra sparse AMWE. Which is like small like we in our AMWE was we had like two variant like one 1B 2B 1B active 2B total 1.3.6. So, we wanted even a smaller one. So, we started like doing dense uh dense like for distillation for like 600 million 300 million and and we started doing like ultra sparse AMWEs like very small ones. Like you know 300 million and 80 million active. And the reason we were doing this is was two reasons. First one is we wanted to be ready for the rebuttal because I was kind of suspecting uh reviewers to ask why because we in the paper of distillation we we also like proposed other stuff not only the architecture like the loss functions the data the way and what's not. I was expecting the reviewers to say okay why don't you apply the same recipe over dense dense variant? So So, like we are let's say in December and we have all these models now. So, we discussed with the with Hook we wanted to train now the bigger perception model AMWE is trained. We have this model already. And now we wanted to train a smaller variant of the AMWE. Uh oh sorry, of the perception. Right? And obviously if you want to do the smaller variant we had like very small let's say dense dense models, right? And we had also very small AMWE models. So, we discussed we wanted to do like this idea of what you call like you know um like AMWE but we have like you know hard routing of image and text tokens. You you choose which which which which experts are for the image and which experts are for the text. So, we discussed and we said okay, that was a Friday I remember. We said okay, let's do do the small dense model because um like like yeah, we wanted to do small dense model, right? Uh in one direction for the image but we have the issue with the small model is the dense model is you have to replicate it for the uh for the for this for the text tokens. Right? You cannot say I'm going to use 28 experts for image and four experts for text. You have to have you have to have symmetry on the architecture. Which means we are we are now back to a larger model. Okay, screw it. We were going to do something very simple. We're just going to train it dense as it is. >> [laughter] >> Train it dense. Like we don't care. No modality specific, no MoE, nothing. We have dense models. Let's Let's plug them. That was a weekend, I remember. So we plugged the 300 million. Okay? So we plug We plug the 300 million model on a Friday to do detection only. And then Monday morning, so we have like around like, you know, 40K steps, which is which was our duration for for for sweeps. Monday, we check the results. And then we see the loss functions are pretty much like very close. I was like, this is not possible. And then we go there and test the model. And like test all the models and what's not and do the benchmarking. And it turns out actually you don't need MoE, you don't need that this like like hard routing, none of that. That's dense works. Like all our assumption that actually early fusion you cannot mix text with image tokens and this kind of was was all wrong. So we had to essentially throw all the trainings we have done for MoEs. We We had to change the title of the paper to signal MoE because now we have dense models. And then [laughter] the perception like now our our model is dense. Which is better for us because dense you don't need like jump, which means our model can work on any GPU, can work on RTX, can work on A100. You don't need H100 for this model anyway. So this is how we ended up having having perception as a dense model. This is absolutely crazy, man. Thank you for sharing. This is This is fantastic. Um and um uh I don't know. Like it it's so cool that like discovery like this happens because like you have some prior and then you go and then you do the things like with like your ideas. But then the the actual result like come back and then you get the Not that I really to check, but just like you get like kind of a glimpse of the truth, right? Of what what is what is actually better for the model to learn. Um it's amazing that you guys came up to to that conclusion uh right in time. >> [laughter] >> Oh, this is great. That was like And then we had And then we had to do Okay, and then after this What was the What was the other question? We We goes As you know, we were training perception and OCR. And we were saying, "Okay, maybe it worked for perception because you don't actually need a lot of text generation. You just All you need is core size sec core size sec. So maybe language is not required here that much as we as we as we think." Okay? Maybe that's why dense works. And then we go, "Okay, let's let's try actually to train a 300 million OCR model." Like the same thing. Which is actually where you need to generate a lot of language. Right? So you need maybe equal amount of modeling both for images and for um for for for language for text. And then we train the OCR model. Same thing. Works. And like works just fine. So which was which was a bit more surprising than it was for perception actually, by the way. So Oh, this is beautiful, man. This is so cool. Um Great. I I have like two more questions. Um well, first like I want to know like actually have two more questions. What's your next plan for for this? I know you're talking about the second iteration of Falcon perception. Do you have an idea about like what what what core things you want to to mix in here or like which direction you want to you want to take this model? Yeah, so essentially like like V1, it was more or less to try out whether natively multimodal works from scratch. Like, you know, we mix everything. And it works in an early fusion manner. And it works in an auto regressive way as well. And all of these three things were like were were were met over one capability, which is, you know, open vocab referring expression segmentation. So now we want to take it a bit further to We want to build a perception engine. Where this perception engine, it should be called by like larger models. It has to be fast and reliable to the point where the same tasks, it does not make any sense to do it for to do it as part of the big model anyways. So we want to go for more grounded captioning. Like captioning where we know where each object that the model mentions is in the image. And also we want to go agentic, which means our V2 has to do be able to do like cropping, zooming, all the operations needed to solve the task if needed. And obviously reasoning. Like, you know, it has to reason like for example the ideas you just mentioned like, you know, like refining the record like having multiple try trials and what's not. So V2 going to be slightly bigger in size. And we're going to expand the territory or the like like the set of capabilities as much as we can focusing on spatial intelligence. We are not We won't be doing any matter any of that. So it will be mostly like spatial intelligence. And you want to push it as as much as we can until until like till the boundary of where we can't do this anymore and we have to call for the orchestrator. Like the bigger models like where we're going to work under them. For example, stuff requiring like a lot of knowledge, reading documents, like doing web search and what's not. So this is our direction and this is what we are building. Cool. Cool. This is great. Very nice. And I can um uh when do you think it would be a good time to start to scale the model um given is like enough compute and given enough data. Do you want to just still kind of stay within like a reasonable uh sub 1 billion token space and then like improve the architecture and then you scale? What's your take on that? Um we don't have like I don't really have the uh answer now because it does depend on multiple factors. So the first factor is inference and efficiency. Because we still want to be like fast, reasonably fast, I would say. Yeah. So there will be like a ceiling like decided just upon this this this constraint. And the second one is obviously the performance we want and how much to model. We start with small and we increase the size only when we see the model is not able to like like absorb the information from the data like anymore. So yeah, we'll see. Cool. Cool. Very nice. Um Okay, last question before we we we close this. Um which research um like finding or area that may not be included in the model yet um are catching your attention at the moment? Uh maybe Yasser and then Folk. Uh Any any finding in particular that you you got very interested even if there's no implementation that are pragmatic? Yeah. I What one of the things I really want to want us to explore is visual COT. Which is something we see with the recent meta new spark. Like, you know, you draw on the image and and what's not and like look there. Uh and do things. Um we don't really see like like the extent to which it it could be like, you know, beneficial. Um it's not really well studied for now. But I think um it's going to be maybe an important block of what comes next uh for for the large models. So I think like like people have explored a lot of like, you know, the chain of thought like the R1 style like reasoning in the text token. What about like reasoning in the visual space? Hm. Um so this is something like that actually catches my my uh my interest. And want to look into it like, you know, next. Cool. Cool. And and Folk, what's your what's your what your interest? Uh for me also slightly related to this about um visual reasoning, but I'm thinking about visual generation as well. Can we also fold in the generation part as one big model? No more decoder, no more quantization. Everything it just lie head like like our segmentation head. And with that everything, generation capability, visual reasoning, they can happen in the same representation space. So that's but might probably a bit far for us to do at the moment, but that is something that general direction I think would be very interesting. And is there any architecture or paper that are going this direction right now or or not even? There is a few, but I don't think anything that's really meaningful like unified. Yes. Cool. Well, guys, thank you very much. That was absolutely awesome. I know it's very late for you, so very big thank you to stay so late to answer all of my questions. It was a lot of fun. Um and yeah, do keep us updated with like the the next situation. I can see from like how you're describing the model that like you're just scratching the surface and like I think you're really onto something. Um so I wish you the best and if you need any help with the augment thing the the the benchmark uh uh like labeling I'm I'm I'm I'm available. There's no there's no need there's no need to worry about it. We we will keep you accountable for that. Thank [laughter] you for the offer. I might do a whole stream just like labeling your stuff, guys. >> [laughter] >> Fantastic. Thank you. See you guys. Thanks a lot for your time. I appreciate [laughter] it. All right, you're welcome. It was neat. Have a nice day. You too. And um that was it uh for Falcon Perception. Um I can't I I absolutely love when you get a like a uh uh a sneak peek about like how actually the last run is is done. Um I think it was something that like Lucas from RC said uh on Tuesday that constrain really do help with research. Like when like I don't know like you have constrain in in budget, in compute, or in time. Um you start to kind of do these move that are more pragmatic but that are get may may get like some very surprising result. But you know that you can't just like meander around and just like look at everything. You have to make decision, right? Um and that's again this this model is uh is a good example of that. Um so was absolutely awesome. Uh I'm very excited to see where these guys are going to like um be able to pop up next especially on the RL side. Um I think it was Vic from Moon Dream that uh explained how he was uh building the model and it was uh it was a lot of reinforcement learning uh that was uh actually a lot of post training. Um so I think like if this model also go and follow a similar trajectory I might need might get something cool. Um the level two issue is also very fascinating at that like it's able to kind of locate where everything is, right? And like count and stuff. Uh but it's not it has a hard time like doing uh the reading. As almost as like they are literate but they're a lazy reader uh in some shape or form, right? Um a bit like I don't know a toddler. You would ask this for a toddler and then they will it will look and like if they don't know how to read you say like hey like pinpoint me the car with the Jaguar and then and then I don't know this other thing. And they'd be like I have no clue what you're saying and then they kind of randomly guess. But if you tell tell to that say to them like like like a pinpoint me the car that is behind the first one, they'd be able to to do it without any problem because like they understand how the world work, right? Um so maybe this model just don't like really like to read. So uh that's also very interesting. Anyway, uh check out the paper, right? Uh Falcon Perception. It's like 47 pages but like uh really well written and then you you get like all this whole story behind. There's also the other two paper the PD bench, right? Very interesting. And then there's the um uh Segelino uh paper about the distillation that they've done that you should check it out. So that is it. Um it was awesome and we'll see each other next week. See you guys.