This Vision LLM solved Time 🙀

Key Takeaways

I asked five Vision language models to tell us time only one could actually do this job this is a classic test to understand how good the vision language models capability is because this is not simply OC or optical character recognition rather you want to do something more than that and why is it critical this is critical because this explains how Vision language model could integrate into our daily life for example if you were to build a visually challenged assistant application one other thing that they might want to do is to identify time so this is kind of a proxy where the models have gotten this could be a bias and training data so that is a disclaimer that I want to start with but in general if you were to understand a vision language models quality there are multiple runs of test that you can do and one of the test that we are trying to do today is give an analog clock or give the picture of an analog clock and then ask the llm to predict what is the time or at least tell us it doesn't have to predict guess the time so all I'm going to do is upload the pictures on five different LMS and ask questions what is the time let's get started first one that I'm going to ask is mixl mistl has launched a new model called pixil and that is available for us to use on their platform chat. mist. so I uploaded this clock do you do you know the time of this if you don't know the time of this let me know in the comment section uh if you are a 5-year-old watching this video probably you wouldn't know this but I guess if you're a grownup you probably would know this okay so the answer to this question question is 555 so this is a Time 555 and pixel says the time shown on the clock is 10:05 see this could be one of the reason um for the high bias in the training data if you go to Google images and then you just literally search for clock analog clock so one of the thing that you would immediately notice is a lot of clock tuned to 5 10 10 and 105 so this could be a uh this could be a bias in the training data that could have like led us to get this understanding but anyways uh the time shown on the clock is 10:05 the hour and is between 10 and 11 10 11 and the minute hand is pointing at five the good thing here is that we have only two hands imagine you added a third hand how this model would get confused okay so the next one is we're going to go to the decorated most decorated llm which is chat GPD 40 what is the time and the time on the clock is 10:30 just outright the hour hand is lightly past 10 uh okay uh the minute hand is on 6 indicating 30 minutes so it's kind of doing the flipping of it so 10:30 so imagine this is this is the hour and and this is the minute and so it has got its own logic I think it just could not understand the length of it so it has got its own logic if I were to go and then say are you sure about the lens and ask a follow-up question U most likely it will guess that it might have made a mistake but let's see if it is you're right um upon reviewing upon reviewing the clock again the ourand does seem to be closer to 11: than it should have been exactly 10:30 okay it tried to review it but it did not get it cool that's good for us so we went to Claude and then we said okay what is the time uploading the same picture 555 the time shown on the clock is in the imagees 11:30 how so it has to give us some detail how is it I apologize for the error and the fact that you asked how Claud seemed to have fixed it so this is exactly the advantage of open ai1 you are not asking how you are not asking it to think are you are not saying are you sure but the model itself is doing that thing and then giving you the response back so in this case it finally managed to get it not zero shot but having a follow-up question of how made sure that the model actually goes through the individual parameters of how a human being would find time and then it gave an answer the next one is a model that I personally attached to for multimodel capability I know not a lot of you are fans of Google Gemini but this model once again it says half past 6 and 6:30 and the explanation is that our hand shorter and thicker hand is pointing directly to six once again um the minute hand is pointing at 12 which signify which signifies 30 minutes P let's say are you sure I've uh just sent it are you sure and uh I want to the model to introspect so you're absolutely right so it uh once again gave me a wrong answer now the one model that got this in zero shot like I asked a question it gave me the right answer back no thinking through no are you sure no followup is mle Mo this is the model that we recently covered it's an amazing model I got to know that what the model that they're running on the platform is a 7 billion parameter model I got couple of comments saying that this model is not good enough for flowcharts and diagrams which is the case for all the models maybe I'll do a separate video just similar to this with flowcharts and diagrams but this model is the only model that managed to get this particular information this model has a very uh different kind of an architecture than typical uh Vision language model it uses open a clip as a back end and Quin 2 7 billion parameter model as a language model the time shown on the clock is 5:55 the hour and is positioned just before 6 while the minute and is pointing directly at 11 indicating 55 minutes past the hour I want to just ask the model are you sure okay just just in case if the model wants to backtrack this is a classical question that you can ask kids and most kids would actually backtrack but this model is not backing track backing backtracking backing backtracking the configuration is very common and easily recognizable position on an analog clock phase so you have got uh a model that can successfully tell you an analog time without having to have any follow-up question so if you were to rank this Mulo comes first the second one I would say Claud is the second one because within followup it at least managed to figure out that there there is some mistake so thanks to Claud for an introspection but the other models fail terribly so I don't know if I followed up did I follow okay um are you sure let's just to be fair and sure I just said are you sure okay it is sure so it said I I'm pretty sure it is 10:05 I'm sorry pixel pixel it is not 10:5 it is 555 I hope there is something useful for you to take away from this video or at least something fun if you have any question let me know in the comment section or if you have got any interesting test for vision language models because Vision language models are going to be the future and we want some Innovative fun way to test these models see you in another video Happy ring