Gemini 2.0 for developers

Key Takeaways

Google just launched the latest and the Gemini family of multimodal large language models Gemini 2.0 flash experimental multimodal models like Gemini 1.5 take text images audio and video as inputs but Gemini 2.0 flash also has the ability to generate responses including audio and images with this ability enhance reasoning skills and tools such as Google search and code execution developers can now build applications with more advanced workflows for example performing multi-step research than visualizing the results with generated code the exciting bit is communicating with Gemini 2.0 using multimodal live API multimodal live API is a bidirectional streaming interface enabling users to have realtime voice interactions and natural conversation you can now interact with the model using realtime audio and video from a microphone webcam or screen the model can handle interruption and spontaneous conversation flow making the experience more interactive now that might be a bit tricky to imagine right now so let's look at some examples this demo shows a user interacting with Gemini 2.0 flash using the multimodal live API in Google AI Studio can you see my screen yes I see your screen it's showing a document with the title demo it also contains bullet points and a Google meet window with a person in the video is there anything in particular you would like to talk about on the screen yeah can you start the demo by just reading the text I highlighted sure I can do that the text you highlighted reads the multimodal live API lets you build realtime multimodal applications powered by Gemini 2.0 flash it features tool use like Google search and code execution function calling and more this demo shows a us are having a conversation with Gemini 2.0 Flash the app itself has been built to work with multimodal live API make a bar graph that compares the runtime of The Godfather and Oppenheimer add the other two Godfathers to the graph pick three random superhero movies and add them to the graph that's a quick demonstration of how you can have a live convers ation with Gemini 2.0 flash notice that you can have natural conversations enabled by the low latency of responses the bidirectional streaming interface allows you to interrupt the model allowing you to refine queries and iterate on discussions so how do you start developing with Gemini 2.0 Flash in these examples you can see the code needed to access the interface this example is written in Python using websockets hi can you list the planet in our solar system sure I can do that for you the planets in our solar system in order from the Sun are Mercury Venus Earth Earth G sat no problem the gas planets in our solar system are Jupiter Saturn Uranus and Neptune thanks next up let's talk about one of the new output modalities Gemini 2.0 flash now generates images and supports conversational multi-turn editing so you can build on previous outputs and refine them it can output interleaf text and images making it useful for brainstorming design ideas or displaying visuals alongside step-by-step instructions image output is available to early testers with a wider roll out expected next year Gemini 2.0 flash features new textto speech style audio capabilities making multilingual native audio output available for developers what's different is that native audio has mind grain control over not just what the model says but how it says it with a choice of eight highquality voices and a range of languages and accents let's hear multilingual native audio output in action what exactly is native audio you're actually hearing it right now everything you hear in this video was generated with prompts like you know this actual prompt on your screen like right now it was all generated by prompting the new experimental Gemini 2.0 flash model like you see in this AI Studio screen cap you can prompt it to just be like dude you know just totally chill or prompt it to speak with oh so very many dramatic pauses so you can see that when Gemini responds it generates not only the words but the style including emphasis tone and natural pauses it can also switch between languages in the same response so I'm starting out speaking English but then imagine how configuring audio output in this way could make assistance more engaging these new output modalities are available to early testers with a wider roll out expected next year so these aren't the only features enabled with the launch of Gemini 2.0 flash 2.0 is twice as fast as 1.5 Pro while achieving stronger performance the model also has improved spatial understanding enabling more accurate bounding box generation on small objects and cluttered images and better object identification and captioning Gemini 2.0 has also been trained to use tools based on developer input Gemini 2.0 can invoke and chain tools iterate over actions and process results to enable more advanced workflows it can be configured to use tools like Google search which can run in parallel finding more relevant facts from multiple sources it can then use code execution or function calling to combine the results leading to tailored responses with more accuracy so if you're interested in trying out these developer tools for building applications with Gemini's new capabilities you can test and explore Gemini 2.0 flash via the Gemini API Google AI studio and vertex AI during its experimental phase with General availability coming early next year if you want to try some of the code examples shown in this video follow the links to the docs and Google collab notebooks as part of the Gemini cookbook to see more interesting use cases of Gemini and Google AI follow the new Google AI for developers social channels there are more demos and examples being announced in the coming weeks so subscribe and stay tuned as always we look forward to seeing what you'll build with Gemini