Cohere Command-R Outperforms Mixtral. Did it Pass the Coding Test?

Key Takeaways

this is amazing now we have command R this is a l language model from C command R is a retrieval augmented generation at production scale it has a strong accuracy on Rag and Tool use low latency and high throughput longer 128,000 context and lowering price strong capabilities across 10 key languages and also you can see model weights available in hugging face for research and evaluation it has a high performance and retrieval augmented generation human reference on Enterprise rag use cases and the dark ping is command R and the light ping is mial you can see the comparison here you can also see another performance here in regards to end to end drag you can see the accuracy for command R is higher compared to llama 270b mixl and gbd 3.5 turbo when used together with embedding and rerank model this is performing higher so the embedding and reranking model is the cair version this lar language model is good in function calling that is enabling access to tools you can see the comparison here between GPD 3.5 turbo mixture llama 270p and command R and you can see the accuracy for the command R is higher this is multi-step reasoning with search tools next if you see multilingual evaluation even for that you can see command R performing much better for needle in the Hast stack test for 120,000 context window you can see the performance here this is nice that's exactly what we're going to see today let's get [Music] started hi everyone I'm really excited to show you about command r a large language model released by coare in this we are going to see about the programming test and also logical and reasoning test I'm going to take you through step by step but before that I regularly create videos in regards to Artificial Intelligence on my YouTube channel so do subscribe and click the Bell icon to stay tuned make sure you click the like button so this video can be helpful for many others like you now we're going to use coar playground and the model we have chosen is command R first we going to use Python very easy challenge so return the sum of two numbers just copying the instruction here the solution is locked so we're going to ask command R to give us a result for this and I got the answer here it was quick now I'm going to test it here and check and it is a pass next let's go to the easy challenge find the discount this will create a function to find the discount so I'm going to ask the L language model to create the function and it is created now going to test the generated function and click check and it is a pause next going for the medium challenge find digital to analog converter function so requesting the L language model to write a converter from digital to analog so now requesting and here is the answer I can see the response is very very quick now testing it here and it is a pause next going to the hard challenge find domain name from DNS provider so this should write a function to find the domain name from the DNS provider so going to ask this to the log language model and clicking enter here and I can see the function got generated so going to copy the code and going to test it here check and that is a pass now going to the very hard challenge identity Matrix so to write a function that takes an integer and Returns the identity Matrix so going to request the Lost language model to create a function and here is the answer copying it let's test that here and clicking check I can see it got paed for four and for the fifth test it got failed so I'm going to copy this error code asking the lar language model additionally I'm going to use the test steps for a better understanding so I'm just going to click Summit seems like it's fixing all those provider test numbers so going to redo it again now I'm going to test it check it's a fail now finally going to the expert level challenge creating ECG sequence copying the instruction so this function should generate a ACG sequence so asking the L language model to do the same and got the answer here just copying it and going to test it here that is a fail so going to copy the error code going to give a final try and the code is getting generated and testing it here so that is a fail so overall this was able to complete up to hard challenge but very hard an expert level challenge it was not able to complete but still it's a good starting point now going to give some logical and reasoning test using GSM 8K data set so Natalia sold Clips to 48 of her friends in April and then she sold half as many Clips in May how many Clips did Natalia sell all together in April and May that's the question I'm going to ask and here is the answer in the month of April it's 48 in the month of May it's 24 totally 72 clips that is is correct so here is another question W earns $1 12 an hour for babysitting yesterday she just did 50 minutes of babysitting how much did she earn going to ask the L language model so 1 hour that is 60 Minutes $12 So for 50 minutes it should be $10 but here the answer is $6 so that is wrong so this is a fail but overall this model is good it is a 35 billion parameter model mainly it is rag optimized with 128,000 cont text length I'm really excited about this I'm going to create more videos similar to this such as function calling with command R rag application with command so stay tuned I hope you like this video do like share and subscribe and thanks for watching