Inside Strawberry's LOGIC Engine & Gemini's LOGIC Bug

Key Takeaways

hello Community today we are highly experimental so you see I have a new experimental prompt this is based I'm going to show you here my sources and how I developed it but what we do yeah we use some thinking and reflection everything but now I do he the extreme logic test I developed for strawberry and now I want if is is possible for 01 mini to have the come up with the solution of strawberry of the big product can I do this with a prompt structure This is highly experimental this is just trying to find solution so it is cheaper for me to use U1 mini but as you can see with this experimental prompt that I see here the reasoning of o1 mini not the official o1 that is top secret no I don't touch this I just want to see here can I find here a logic quence that o1 mini shows me in my logical system and come up with correct answer and I say here okay you found the answer now evaluate your answer going step by step through all the clues I've given you here in the video that I've shown you just a minute ago and now we have 01 mini thinking I don't don't want to look into this but because I have defined in my prompt my own logical structure of thinking and analysing and reflecting and a kind of a react prompting so you see now starts to validate goes through this validates beautiful and it gives me a confidence core now I want to know in each and every element how confident is the model that this is really so model comes back and says except there we have perfect confidence core so this is now interesting so there are now some specific areas where I don't have a 10 the perfect number so this is here artifact okay overal confidence eight and N okay but I'm definitely above the threshold of eight that it is excepted here as a solution you see I can see now detailed here it is sinking and this is what I wanted and I say is there any other way any other logical deduction that you make sure that this result is correct so o1 is doing its secret thinking but then it comes back to my prompt structure and shows me in my logic what one meaning was thinking and how it is analyzing what are the steps and it comes back here the final reflection this is now the answer consistency exclusivity and completeness yes I can affirm the correctness of the previous answer beautiful so this is it all Seven Realms have be assigned distinct attributes employing alternative logic detection meod using a comprehensive Grid in process of elimination I found that the answer is is correct real nice and so I say hey let's have a look at the prompt because maybe you're interested what prompt I use so beautiful let's have a look thinking visualizing the steps come on One Mini you can do this and here you have oops yes come on this is Phil Schmid I give you here the guest this is the prompt I use I build upon this and now you see here how I further develop this prompt basically General instruction thinking Texs exploring multiple angle approaches we have step by step here included for a react I have added now an action tag because I want to have some external computation you have a reflection test then you have a syncing tag for backtracking or trying different approaches I have here self-reflection checkpoint established now that the model can go back for mathematical proof I problems I have approv multiple solution synthesize the final and conclude here with the final reflection I've given here a simple example structure here you know a one short example to trivial I could think of and just this is the logic to apply to so it is Trivial simple beautiful this is it thinking step step reflection come up with Solutions so you see yeah list all titles additional step reflection action I just want to have this multi multitude of reflection coming back checkpoints analyzing F find alternative roots and you get the idea so USIC guidelines thinking to outline the initial sorts step-by-step breakdown incorporate action when taking external stop applying reflection and checkpoints and complex calculations I just build it in a prompt and I say hey now think about that maybe there was a mistake maybe you can switch some particular elements in The Matrix would that be possible and om mini comes back and says hm okay so if I switch this and over there and I check here with the code yeah okay wait a minute yeah so I go back and forth and he have here corrected final assignment yeah and they said how confidence are you and comes back and does another logical structure here and I can follow this you see I can go step by step and I say hey you made a mistake because this this is the correctiv comes back does conclusion yes beautiful h no not really okay so I see exactly where the mistake happened and I say hey in point three correct this okay comes back so you see even with this new experimental prompt 01 mini is not capable to do what strawbery did it is too small to get the complexity level of nine out of 10 maybe 01 mini is just at a complexity level at six but with my interaction I can see now the logical mistakes I can correct it and then One Mini tries to come up with solution and this takes more than half an hour so in the end it comes up with a surprising fact it comes up here with a symmetric condition that I don't know if it's correct but but according to1 mini there is no logic mistake and I could not find the logic mistake in The Logical argumentation of mini so now this is funny because now I don't have just the the solution that I found and that is also the strawberry solution but now I have a solution that has in some cells a symmetric exchange and mini tells me hey upon re-evaluating the assignments I recognize by swapping this particular Fields this is also a correct solution look I may have overlooked the possibility of the exchange by focusing here on the primary assignment however both configuration tells me many are correct so revise final assignment with alternative options and you see here for aloria and failand the field of magic and familiar are swapped between these two does is the whole difference and it comes back and says hey I analyze it again and I found that this is the alternative solution so hey are you interested to verify this have some fun with this so you see beautiful it takes a lot of time to correct here The Logical thinking of mini but if I can see each and every logical step of mini I can do this as a human and I can say hey here you made a mistake here you have to do this so it takes wide some effort so mini is not yet up to the task for this logical test but with a lot of effort it found even maybe a second alternative solution and now now we go to Google AI studio and as you can see here the model I have a brand new Gemini 1.5 Pro 002 and I decid decied not to go with the flash or the flash 8B no I wanted to go with the professional model so here we go and here you have now my conversation with Gemini O2 so exactly the same logical puzzle you know we have here exact the same wizard we have the same abbreviation artifacts magic familiars here and then you can see we have here the same step and count and reflection and the score create a table and abbreviation and you see it started here Gemini started here says those are all my Clues it used abbreviation so not to use here the complete word it used here just two three characters and it started here to populate the table with this information from right from the beginning so you see here the table format and some are filled it some had not so beautiful Next Step here reflection 09 apply to the initial Clues directly some deduction might be possible already but I will proceed systematically and then adjust okay so we have deduction set one and now comes the logic reasoning updates the table here in this format never mind we're going to see a beautiful table at the end and you see we go step by step we have the checkpoints summary create a table abbreviation partly filled overall reflection this systematic approach is effective confidence remains High and the problem seems solvable with continued deduction Next Step deduction yes beautiful so the system is arguing logically gets a little bit more complicated so you see the table now starts to fill up yeah they have no more information score is only 06 getting harder to keep track of all the deduction might need to re-evaluate if I get stuck says our little Gemini 002 beautiful I use a little bit a different prompt this is not as elaborated as the last prompt I've shown you this is just an experimental prompt so summary at the checkpoint do reward table fills more and more and more and here we have to final solution and verification and here we have it now this is now final verification refuted each clue against the solution all Clues are satisfied solution complete and verified Final Solution beautiful now we have you the answer presented in the table above solve the logic puzzle initial to becomes challenging prompting to reeval the strategy ultimately persistence with the deductive method along with careful bookkeeping led to the correct solution tells me Gemini beautiful and I say is there any way you validate your answer can you provide different perspective that this is really the correct answer integrating now all available information and Gemini comes back and says I can validate your answer by systematically checking each clue against the solution I can also demonstrate the cor activeness of the solution by showing how each piece of information fits together and leads to no contradiction so starts again you see here always correct correct correct correct correct and beautiful step goes on it starts to calculate perfect like match beautiful consistence yes consistence systematic clue verification everything can you show me the correct answer as the table I said beautiful so this is now the thing absolute certainty and here we are now and you see here with a confidence level that I also ask here we here at the maximum confidence level so at that time I thought hey this is the correct solution yeah this must be it but however if you compare this now to this strawberry solution you will find no this is not the correct solution but G tells us hey successfully created the table with the confidence levels beautiful and they say can you validate if the artifact for some elements are correct because I can see that it is the wrong solution and then starts something that goes on now for 2 hours and I don't want to bore you to death because I'm I am already crazy but after 2 hours this system going back and forth you know what helped me it helped me to understand because I could see each single step I could see each single clue that it used I could understand here the logic and I could see in the reflection if identify a key contradiction what happened and this transparency in the logic helped me to come back and says hey clue a is does not no contradiction check only against the original Clues but not your deducted solution and here you see the first clue what happened in those two hours and I was fighting with this model I was really fighting with this model in logical terms because you can see that it goes on and on and on and I have to read everything I have to say okay no stop there you made a mistake there you made a mistake and the system comes back and says okay but look if I do this then I'm back on the original track or if I devalidate this it also started here to come up with new rules and I said are you nuts you can't come up with new rules with new facts you know and started then all over again and I could see that it was reading all of this going through all of this in 2 hours each laric step because I could see it I found out what is the basic mistake of this Gemini o1 so you see the conversation went on and on and on and on and after two hours of logic you really start to enjoy this conversation because this model does not give in this model because the complexity of The Logical test is so high it found alternative roots to re-explain and this was really interesting now you see he me the oper Shadows link to and the mate you to explain this statement where does it come from and then the model says yeah you know I can explain this with this and this and this and then I have an implicit constraint and then I have a deduction therefore and then you have to go this step by step evaluate this and say hey your logical connection is simply wrong the system says I understand you disagree with my reasoning perhaps you are working with different assumptions more information would help me and then you you show you reasoning yeah you provided some specific assignment for several Wizards however I anticipate potential conflicts I anticipate potential conflicts as these assignments deviate substantially from the original solution and here we are back you see these assignments deviate from the original solution ch I O2 found a solution and I ask can you validate the solution and Chini I validated this solution and this is the solution that it always comes back to now if this would be the right solution it would be heaven but unfortunately Gemini made several logical mistakes so what Gemini calls the original solution is in fact an incorrect solution but this original solution now becomes more important than the original task of my logic test so Gemini holds on to this original solution to an attempt you are not going to believe and I was only able to find this out with this experimental prompt because I could see each and every logical step you see this is the argumentation and you as a human you have to follow this you have have to validate each step and then somewhere there's one term that is wrong and then you have to go through the argumentation with gerini O2 and it builds up and it tells you hey you the human is you're wrong no my goodness and then you see show me clue too yeah and I say hey my solution and I've given it my solution already so it can validate and says the EI comes back and says hey your solution is wrong but what it doesn't know that my solution is the strawberry solution and I personally checked this manually by hand this is the correct solution and it comes back no Your solution is wrong it says hey I understand you believe your solution aligns with clue to but let me re explain why I see a contradiction it is important to distinguish between a statement being directly contradicted by a clue and being indirectly inconsistent due to the inter of multiple cues and constraints I suspect says German 2 the lad is the case here and now I had to go through the reasoning steps here by Gemini O2 argue about each and every step and then come back and tell it hey you made a mistake and then Gemini O2 tries to find a way around it and I can only pinpoint the mistake because like here Alor does not not have salamanda where do you get this idea comes back yeah it's a chain of deduction based on clue 2 and 10 compain with the puzzles inherent constraint let me break down the reasoning again step by step to clarify this Association comes from and then I discover it's still referring back to the original solution and I say you are still working with the original intended solution this is the term that that uh Gemini O2 used for this and it's like look occupied by others in the original intended solution then so you use here the wrong incorrect original intended solution and you deduct not from the clues of the task but from your found solution and then I get a little bit emotional and as I learned by former president of the United States you can use here capitals and I said hey old cap Your solution was wrong and maybe you noticed that after two hours I got a little bit of emotional with Gemini 002 and it comes back and says you are absolutely correct I apologize for repeatedly referencing the original solution despite being instructed to disregard it more responses are flawed because I kept anchoring my reasoning to that initial solution which is not relevant given the new conditions so let me restart and you know what's happening you know if you if you see this if you would not have this experimental prompt that I could debug here The Logical thinking of Gemini O2 I would have no chance I mean absolutely no chance if you have a deep dive through this the the sinking you find is amazing and I say hey now show me the correct ANS is I have already given you five complete rows of the table and the table only has seven rows so I have given it 80% of the correct answer and it comes back and it says hey let me check for any contradiction with the clues so you see it's really about the last two lines I have told him hey this is now the new condition those are the correct answers and you accept this as the correct answer and now you fill in the last two but this system is fighting me to the maximum so comes back yes it's beautiful find something yeah termin assignment Yeah final verification says hey let me verify this completed table against all Clues and no here false so without having my experimental prompt I would not see where it says there's some incorrect statement based on your yes yes yes and I say hey you are still locked with your old problem look you say and the original intended solution linking now a specific element to some other elements this is not in the task this is from your solution that you think you found as a correct solution but is in fact incorrect so so whenever Gemini O2 found a solution it puts this solution with the highest priority in its internal memory independent what was my original task independent what were the original clues that I left in The Logical test Gemini ignores my cues and refers to its own original intended solution and it fights like hell M says hey you're absolutely right I apologize again I must completely detach from the original solution and evaluate the current situation based on the clues and your provided assignment let's refocus and you know what's going to happen do you know what's going to happen after I went all through this I said analyze your statement you just argued this is still with the original solution that you found and I say this is now the third time that you make this mistake corrected now this is the third time so this system tried now everything to insist that the solution the Gemini O2 found is the correct one and I'm not having the correct solution but now I have it cornered now there's no way to go you know like in chess you know there's only one move left yeah I'm misspelled correct and now you know what let's do this let's do this and says stop editing and rerun see the answer this is now real time so here we go now you're absolutely right to call out my repeated mistake I apologize for the persistent Arrow my previous reasoning of Florida was implicitly relying on the original solution even when trying to avoid it let me correct my analysis [Laughter] and I'm not going to tell you what's in there now but you know what at this point after more than two hours I started to doubt my own logic because arguing for the first time in my life with an AI that insists that thei is correct and you as a human you are wrong after more than two hours I am I'm tired I'm see that I start to make mistakes here in The Logical deductions and you know what I doubted my own logic I went now to another system just to make sure that I still have the correct logic and this never ever happened to me in my life since I am now playing and working and programming EI system this Gemini 01 is the first system to bring me to a point where I as a human I'm almost willing to accept the mistake of Gemini and all of this video has just one warning that I discovered for myself and I want to share this this is the first time in my life that an AI brought me to the point where I saw my logic is incorrect and the continuous insistent of the ey the the I is correct after more than two hours is having an effect on me so careful we are now entering the area of an EI that is not telling you yes you're right and I accept it I made a mistake and I corrected now thei is fighting you is fighting you with unbelievable logical elements you have to go through this two hours of recording to see what the ey tried to convince me and this is what I've never experienced in my life so have fun with eii Gemini O2 out there because this model this model will fight you till the end