AI Characters: Animation, Consistency, & Video!

Key Takeaways

let's talk about what's buzzing around in the world of AI next week is nvidia's GTC I will be in California at GTC so if you're going I might see you Nvidia is also giving away this RTX 4080 super or I should say me and Nvidia are giving it away and it's completely free to enter so check the link in the description if you want to see a video on how you can get entered to win this bad boy and right now Sora is trending on Twitter not just because open AI continuously releases more and more very very impressive demos of Sora in action just a straight text prompt to video but also in a recent interview open ai's CTO gives very vague and strange answers to the question about what data they used to train Sora before we get into it I do indeed have a quick word from our sponsor let's be honest guys as AI enthusiasts we browse a lot of websites that aren't exactly welln hey we're always checking out the most interesting new AI technology whether it be from a small startup or sometimes even a completely unverified website that's why I'm happy to say that today's video is sponsored by none other than nordvpn yes the best VPN in the business if you go ahead and click the link down in the description below nordvpn is offering just Matt vidpro viewers a special offer four extra bonus months on top of their plan for us AI Tech enthusiasts nordvpn covers a wide range of capabilities first of all connecting to nordvpn is dead simple it's an automatic quick connect if it's installed on your device and they've got over 5,300 servers in 60 different countries meaning and this happens a lot if there's a certain AI product that is not yet available in your country and you want to use it you can use nordvpn to Tunnel through and connect to a country that does actually have access thus allowing you to get access and again connecting to some of those let's say unverified AI sites they have a double VPN for Extra Protection and automatic threat protection to ensure that any AI tools you might download on your own computer are safe what's also really cool is nordvpn's entirely risk-free if you don't like it well you don't have to pay for it they have a 30-day money back guarantee again if you click the link down in the description below you get four free months again I really want to thank nordvpn for sponsoring because this channel would not be possible without its sponsors and of course the viewers watching I promise you won't find a better VPN anywhere else we'll go ahead and take a peek what data was used to train Sora we used publicly available data and licensed data so videos on YouTube I'm actually not sure about that okay videos from Facebook Instagram you know if they were publicly available um available yeah publicly available to use um there might be the data but um um I'm I'm not sure I'm not confident about it what about shutter stock I know you guys have a deal with them I'm I'm just not going to go into the details of of the data that was that was used but it was publicly available or licensed data obviously it's been one of the main concerns for AI lately legally who owns the data that was used to train these models and how does the copyright work on that the question definitely seems to make uh the CTO Mira moradi here very uncomfortable people are pointing out that it's kind of concerning that she doesn't know where the data exactly came from but I think you have to keep in mind that the way that they collect the data for these models is definitely not handpicked so it's probably some sort of web scraper that goes and grabs publicly available videos along with you know as mentioned licensed data this AI stuff is pretty new so it would make sense that we don't really have a specific copyright law in regards to this but since Sora is just for this internal research use it does give them a little bit more leeway than if they were using Sora as let's say a product that we could all pay for and utilize I think there's a high probability here that the CTO was nervous because in her mind maybe she thinks that the interviewer is attempting to pull a fast one and trick her into giving an answer that can be you know some big headline and get a lot of clicks for the Wall Street Journal let me know what you guys think down in the comments below I'm not a lawyer so I don't know the legal circumstances on what kind of data you can use but I'm sure that open AI has lawyers and wouldn't just go off to train a model without ensuring that they won't get absolutely destroyed in a lawsuit but maybe I'm wrong so now I want to dive into anthropic ai's Claude 3 ha coup which while not as powerful as their best brand new model which coincidentally in my opinion is the best large language model you can have access to is the fastest and most affordable model in its specific intelligence class here's the thing about Claude 3 ha coup this model competes with GPT 3.5 input tokens are a staggering half the price while output tokens not so much at 125 in comparison to 150 on the gpt3 side what's cool though is it is better in every single use case or at least every single one of these benchmarks especially in sectors like grade school math where we see an 88.9% on zero shot versus only a 57.1% on five shot with g GPT 3.5 and not only that this model has Vision capabilities as well meaning you can upload images to it and ask the AI questions about those images or use those images in creative ways where that's just not possible at all with GPT 3.5 there's no vision capability whatsoever with that model for context it also beats out Gemini 1.0 Pro in pretty much all of these bench marks as well the thing about ha cou though is that it is three times faster than its peers and they mentioned that this enables Enterprises to quickly analyze large volumes of documents such as quarterly filings contracts or legal cases and it's interesting that they list these as the examples because this is something that I don't know if I would feel comfortable trusting one of these smaller Hau models with however perhaps Hau has really good recall just like its bigger brother models meaning you know if you upload lots and lots of documents it's going to be able to analyze it without hallucination so this also happened as well open AI partnered with figure to give this robot more or less a brain via API access to open AI models and it also has Vision capabilities as well via open AI which is pretty darn cool figure more or less obviously has the robot itself here but they also develop neural networks that are delivering fast low-level dextrous robot actions and as you can see in this little video right here while this little bot has the ability to just put stuff away right in this basket or you know put some dish away all while having conversations with the user and obviously it can see when a task is complete because it has those open AI Vision capabilities so this right here folks is your closest Glimpse or one of your closest glimpses at what the future is going to look like with robotics and AI combined I think in the next 10 years or so you're going to see some massive developments and improvements in regards to the combination of AI large language models Vision models Etc and Robotics you're going to see Rob robots working at different businesses you're going to see robots inside of people's houses you're going to see them driving people around for sure except the cars themselves will be the robots more or less I've been talking about this a lot lately and especially seeing all the autonomous AI agent stuff I just think the combination of autonomy plus the robotics here is really something to behold it's really going to bring some of that like futuristic sci-fi stuff to actual life right before your eyes I can't even can imagine what the world is going to look like in like 30 or 40 years just absolutely mind-blowing Brett adct here who is the founder of this company noce that there's no tricks with this video it's all happening via neural networks and that the video is not sped up or edited in any way he notes that they are approaching human speed with these robotics apparently these cameras are feeding into a large Vision language model by open AI I don't know if it's gp4 turbo or what it's some sort of vision model it could be even a new one that we haven't come across right now the cameras that allow the robot to see are operating at 10 htz which honestly isn't too bad definitely enough to pick up a lot of what's going on I don't know you guys can think this is creepy or cool it's probably a little bit of both in reality so next up here let's talk about Devon I made a video on Devon yesterday it's this autonomous AI agent and appearing from the videos that cognition Labs uploaded of them testing and using Devon it's the best AI agent we have ever seen to date McKay Wrigley here is not a part of cognition and he got access uploading a full 27 minute video of him using it in real time here on Twitter which I'll link down below he notes that it feels like the chat GPT moment for AI agents and that overall is just exceptional work from the cognition team now I can't really get the video to play personally right now but like I said it'll be linked down below and the fact that they're giving people access who are not a part of the team and who are just going to make you know some Twitter posts they're not paid to review this stuff I think bodess well for how well Devon actually works as a real use case I've also dm'd them to try to get access myself so we'll see if I end up getting access because I'd really love to put this thing to the test it's important to note open AI probably has some gp5 on the way that hopefully will utilize autonomous agents and I've said this before I think that it has to utilize autonomous acting agents in order to keep up with the rest of the industry as is clear with stuff like Devon in thropic Claude 3 Opus also has the ability to use AI agents it's just not public yet so we'll see how that works as well it'll be a very very interesting year and I think it's going to be the year of AI agents this also popped up on my feed and I found it very interesting this is cartwheel this is a text to animation AI tool and as you can see from this little video demo you can just text and it will animate your character in whatever ways imaginable I have a feeling this only works on human characters right now because how could they you know abstract this for frogs or dogs or other kinds of creatures oh and by the way it's actually called cartwheel that's what this AI is called their system here is still in Early Access as Andrew notes who used to work at Google AI and open AI this is pretty darn cool though this is walking on a tight RPP and trying not to fall and you can see it's able to animate this character in a very realistic way I mean it looks like you did map on someone for sure that's very impressive but I bet it fails quite a lot right now as you can see any character is really applicable to this practice boxing punches okay wow this could be a pretty big year for AI assisted video game creation animations sound effects and coding with Devin I also wanted to touch on Mid Journey's character consistency here in this news video dogen Earl fantastic AI account here on Twitter by the Way shows off this example with Sydney Sweeney uploaded as the input character for the character consistency and it does a pretty good job at replicating her in these various generated images it's called character reference and there's a few ways you can use it you just do dash dash CF with the URL and then you can do D- CW to adjust how much strength from Zer to 100 zero being the face only and then 100 being the full character and manual here on Twitter which is also a fantastic person to follow for AI news also happens to be the founder and CEO of scenario. which was very much working on consistent characters gave this a test run as well he notes that he doesn't think it beats a good fully trained Laura model yet especially when paired with IP adapter showing off the comparison of the mid Journey feature with the lur and IP adapter and just Laura with no reference image and honestly I would have to say that these results are pretty telling while I think the mid Journey feature is absolutely a phenomenal start it gets the blue skin the pointy ears the very bright yellow eyes and the overall facial structure it doesn't look like the same exact Goblin creature as you see over here in the Middle with the lore and IP adapter I mean this one I would say is pretty much perfect every single time getting nearly all the same exact features eye shape is correct the facial bone structure is correct every single time and the skin color is just bang on you see Laura with no reference image is also really good as well so it doesn't beat the traditional options but it is a heck of a lot faster and easier to use than you know setting up a whole Laura of course this will all be linked down below if you guys want to take a closer peek at this I think it's really fascinating though and we're definitely going to see the era of consistent characters generated with AI this year I can't wait till stable diffusion 3 releases as well because this will just make all of this more coherent as well with your prompts you'll really start to be able to use this for real world scenarios also something I wanted to touch on really quick this week Gemini 1.5 is becoming accessible to people I actually got access a few of my Discord members got access as well sort of in this like early stage Ruben HED here on Twitter notes that it is not a really great model for the high token limit that it is advertised up to first test that he did was read Inception to write Inception 2 50,000 tokens for the whole script and it's good at setting an outline for a sequel to Inception but eventually around the 13th chapter or so it starts to fall apart and begins to hallucinate over time meaning it's not very consistent with the amount of tokens that it can intake at once so it's just really not all the way there and well it seemed very impressive when Google announced it it doesn't seem to have the chops to back itself up I also tested it with some multimodal video which it did okay at but it just didn't have a level of clarity that got me really riled up or excited like I literally sent it one of my own videos and it did okay getting a synopsis out of the video but it started to hallucinate some weird details it wasn't all exactly as it happened in the video and finally everyone is sort of anticipating this open AI model release everyone on Twitter is really hyped about it even though open AI hasn't really hinted at too much but everyone's sort of expecting a GPD 5 because a year ago today we saw the announcement of gp4 and GPT 4 was a really huge day I even said happy birthday to GPT 4 because I think it was that important honestly to the world of AI it was a model that was very far ahead and is still competitive a year later and year in the AI space is a very long time so we'll see how things go again I'm going to be at GTC next week and I'm going to be covering lots of AI announcements and cool AI Tech that I might see there so stay on the lookout for that and also enter to win that RTX 480 check out some of my recent videos and I'll see you in the next one goodbye