Cohere API Community Demos | October 2022

Key Takeaways

[Music] welcome everyone to our episode of collab Friday after a break we are having a comeback and we are having some exciting updates when it comes to the format of the event so I am super super excited to have you all here thank you so much for taking the time to meet us whether it's your morning your afternoon your evening just to give you a little bit of an intro I'm Sandra I'm looking after go here community and I will be your NC today and we'll be working you through our agenda we have planned some really exciting stuff for you today hopefully you enjoy it uh so first up we we are going to invite to the stage Nick Frost who is coh here's co-founder and he is going to share coher Hot Topic of the month before we share that though um I wanted to let you know that we're super happy to share that we have actually surpassed 2,000 members on Discord community and we're super super excited about that we will be coming with a little well maybe not such a little cookie uh and thank you gift um for those of you who are the most active Discord members so stick around and next week you should know what that is thank you so much for for being with us we are growing we are getting more and more creative we are warming up in terms of building stuff and collab Friday is also a platform for that so to give some Spotlight to your awesome ideas So yeah thank you so much for being part of this coming back to the agenda in a second Nick is going to share a few words that are relevant to our community hopefully so stick around and see what he has installed go here Hot Topic will be a monthly segment where we will be sharing some exciting stuff around our community some interesting conversations that are going on or some product updates next up we will be hearing from our demo presenters after that we will be going into breakout rooms one breakout will be for the tech oriented folks one for the business oriented folks I mean folks with business oriented questions SL Tech oriented questions and we will be taking conversations there and we will finish up with Aiden's pick of the month which is this new thing we did with Aidan where he looks at the demos that Community folks created and chooses his favorite one within the particular month we will be talking more about that in a second and we will be closing with um updates on upcoming events so stick around and see what we have in store so yeah we can go into coher Hot Topic Nick are you with us yeah yeah I'm here I'm here awesome I'm ready to present coher Hot Topic uh I didn't bring any Funko Pops or link 182 t-shirts but I think I can still do a good job of it uh so hi everybody I'm Nick uh I'm one of the co-founders here um today I'm just talking to you about Discord Bots that's the thing I wanted to talk about um and the reason for that is I have built a lot of Discord Bots with coher over the past year um as a co-founder here I spend a fair bit of time just like building stuff with our API uh and I do that in part because it's good to know what we're actually building and in part because it's really really fun uh and Discord Bots has been like a place where I have uh often like gravitated to for demoing this stuff and the reason for that is there's a whole bunch of stuff that that Discord does a really good job of um that makes using coher like a lot easier um so for instance like Discord is already maintaining the state of a conversation so you can always just go back and read the information that provides an easy way to trigger Bots um and it's really easy to share like once you've made a Discord bot you can put it into a server um that you maintain and then if you want to you can give a link to people and they can put it into their server so it's a really easy way to like whip something up you have the whole interface already for you um and then and then you can share it and demo it to people so there's two right now Discord bots in the uh in our Discord uh and if you go there go down to a section called uh project demos yeah in Project demos there's a there's two right there and there'll will be more soon so the one the one that I wrote is called the grounded QA bot and that's a thing where you can like ask a question um and then it will try to contextualize that question Google search the answer and then find the relevant information and give it back to you in the in a natural language sentence um so that one's really fun you can you can ask a question in the bot and then if you Emoji react with a question mark it'll try to answer the question you can also DM the bot and just use it as your own personal conversational search agent um and if you click the bot there there's a link and you can add that into your own server um the other one that's being demoed there right now is the web LM and that's the thing that Aiden and now a few other people have joined and are working on and that's a bot where you try to you try to it it like automates a browser through language So Yesterday Aiden was able to successfully give it the command buy me uh I think was like buy deodorant from Amazon and it was able to like with that command Google search Amazon Google search deodorant give the thing and with like minimal interaction from him it has shipped deodorant to his house so that's great news for for him and for his co-workers uh and also also for coas being able to do useful stuff um but the thing today that I wanted to tell you was that if you guys are making Discord Bots as demos and you made a Discord bot that you're you're proud of or you want to share you can message us and we'll add it into that demo section so we'll make a channel for it we'll give permissions to that bot only for that channel so it won't be able to read anything else um but it can be it can live in that channel and you can demo it and share it and share your work with other people um so if you make something that's cool message me or Sandra or anybody else and we can give permissions to the bot so that it can exist in one channel and people can see it and then if you want to like get other people to work on it like Aiden has done with the web LM stuff feel free to share the GitHub and and see who else is working on it um one note just for the future of Discord Bots right now they like they used to be kind of brittle um but they've switched over to like a version two of the Discord python API which is mainly what I use and that is really focusing on slash commands so normally you would trigger a bot by like writing a special sentence or something like writing a a token at the beginning of your text but now there's a whole framework for giving it commands um so the ecosystem of Discord Bots is like evolving pretty quickly um and there's it's honestly getting a lot better now you can like do slash command and then that'll give you like a an input window and so you can make mod you can make like input windows for people to interact with the bots so yeah great time to be building them if you build one we'll put it in the we'll put it in the demo section um and if you want to check out the demos that are already there feel free if you want to contribute to the webm thing that's that Aiden is working on that's fine in my grounded search spot will be open sourced next week so if you want to contribute to that after next week that'd be great too uh yeah cool any questions on on that no cool then I'll pass it back off to you Sandra thanks for thanks for having me awesome thank you so much I will just add that um if you're considering turning your demo with which you're are applying to community demos event um to turn it into a Discord bot we have added this little section of a question where we're just asking you if you're interested in that if you indicate that we will also reach out to and help you turn it into a Discord B that will be displayed on our server which is sweet thank you so much all right so we will move on to the demo section um we have four very exciting Demos in store for you today um after each demos there will be a feedback and Q&A time and we have coher mentors today with us uh David Stewart Robin Gyer and Adrien morisot who will be giving life Fe back to um our lovely makers however if you have any comments or any questions around the particular demo that you've just uh seen just feel free to either let us know in the chat section or unmute yourself and go ahead and you know share it directly so without further Ado first up is Arjun Patel with prompt engineering assistant for image generation arjin are you with us yes hello um can you hear me yes we can hear you go ahead all right I'm going to go ahead and share my screen and we'll get started can everybody see my presentation beautiful hi everybody my name is Arjun and today I'm going to present my project called perfect prompt before we get into that I want to introduce a little bit more about myself I'm a data scientist with experience in structured data I was uh formerly working at a company called appin and another company called SPO which I worked on building a speaking coach for people kind of like dualingo is for learning a new language or grammarly is for fixing your errors when you're typing speak was meant to be an application to help improve your public speaking skills um I then moved to Industry working at a company called appin and I was laid off promptly after working there and I had a lot of time to myself and I realized that there are a lot of really interesting crazy things that are happening in the space of natural language processing we have a lot of foundational models now that can be fine-tuned or we can use transer learning to do a lot of interesting Downstream tasks on and I wanted to learn more about those things so I started learning about Transformers and I came across a really weird problem that I came that I learned about and that problem was related to image generation I was learning more about image generation how people create prompts and stuff and I realized that the current free ways to do it were very annoying and very timec consuming so on my presentation here I have the hugging face Radio free demo of using stable diffusion and it takes about 43 seconds for you to generate four Images given a certain promp and to me that felt really hard that user Journey felt very uncomfortable because you would have to create a prompt you would wait 43 seconds for that prompt to come back you would think about what you want to change with that prompt and then you repeat that whole cycle over and over again so to me this felt very time consuming there are options to make this processing faster but then you have to spend a lot of money in order to host the model yourself or do something along those lines so that felt very expensive and frustrating to me so I wanted to see if I could build a product that could resolve this issue to shorten this user journey of creating prompts and images and once I did that I built something called perfect prompt with coher apis it's meant to be a One-Stop shop for prompt engineering and image generation and today I want to show you all how to use it and what we can actually do with it so I have the tiny your all here if you want to work independently but I thought it would be really fun for us to come up with an image together with these few categories in mind so in the chat if people have any suggestions for what we should generate I'll show you how perfect prompt Works in order to create suggestions for you in order to get better at that image that you're working on so the categories that we have right now are architecture Cottage core steam Puck cyberpunk laps Landscapes and watercolors and perfect prompt will try to match your prompt to one of these categories and then provide suggestions for you to improve that prompt so here's what the web app looks like if anyone has a suggestion for a prompt feel free to type it in the chat or tell me otherwise I have a prepared prompt we can mess around with oh beautiful Robin thank you that one looks amazing brutalist kindergarten uh kindergarten playground so we'll hit sh enter and the model will start running this will take some time and what it's doing is it's embedding and trying to classify this prompt into one of the six art styles that I presented before once it figures out what art style is actually happening it will seed that keyword into a coher generate API along with a phrase that says hey take this prompt and rephrase it with more detail and it'll generate 10 different suggestions for you to use so once this stops running we'll have a bunch of different suggestions for us to have inspiration on in order to create our next prompt not only that I found an open source implementation of stable diffusion okay so this this is our little image that came up it's really weird looking it does remind me of the days I spent staring at brutalist architecture and let's see what suggestions came up here so we go down here and we have our cache and we see some really weird stuff thank you Robin for your suggestion we see light at the end of the dark or at the end of the tunnel which kind of reminds me of concrete uh we see some other keywords that are important something hidden HD 8K intricate elegant abstract all of these are really interesting because when someone tries using an image generation model for the first time they're not going to think of phrasing their prompt this way but the generate API allows you to think about how to phrase your prompt in a certain way in order to get it to generate the images that you want uh I'm getting close to the end of time here but the idea is that you can send back another prompt and make more images as you need and yeah so thanks for listening to my demonstration if you have any questions let's hear about it I'd love to talk about how my product works awesome thank you so much for sharing um we are opening the feedback and Q&A section and the tiny URL has been pasted in the chat so you can go and test it out yeah can we hear from anyone's feedback I can chime in super impressive I think it's a very cool implementation of using generate in a lot of the cohere stuff um to get the prompts initially were you just using your knowledge and familiarity working with these image generation models where you like you were giving it inputs and examples of sentences or with like a broader data set you used yeah great question so I was able to use a website called lex. Art which acts as a search engine for prompts for stable diffusion they have an open API where you can pass keywords to their search engine and you can pull down like the top 50 prompts that come up associated with that search engine so I used the categories that I had to scrape those prompts from the search engine then I fine-tuned a coher model on the prompts from that in order to create suggestions very cool we have a question in the chat from Josh I saw uh so the weight for the output is for a few things it takes about 3 seconds to generate the image using an application called replicate uh we're not interacting with Lexica during the runtime of the application the rest is just using the classify API and the generate API so I think the rest of the lag is from those two apis I tried implementing a speed up where instead of retraining the whole model by sending examples into the classify API I fine-tuned a classify model and then tried using that but I only got a little bit of a speed up so that's something I'm working on thinking about awesome any business oriented comments around monetization and stuff yeah we can hear you okay right um I find it quite interesting because I came across on um open CD they have a a weekly webinar and um they have a competition now where they give you a a picture and they actually to come up with the prompt so I think it might be interesting for you to apply that and see how it's you know is able to the problem that be interested in application that sounds really fun actually one of my first ideas for like an application was to have a game where you present an image to two people and they both compete to get the prompt and that way you would yeah that way you would get a data set of like okay what do people think creates this thing and what's the actual thing is but that sounds like a lot of fun I bet you could use this application to get the prompt that you think it actually is faster interesting all right folks um you can save your further questions or feedback for later and also for the flup discussion um let's move on to the next Dem presentation thank you so much AR that was sick we loved it thank you y thanks everybody um all right next up is Toby Toby do have you here Toby is presenting project topic generator for engineering students over to you all right thank you very much uh okay my project title is project idea generator for engineering students so a brief introduction of myself um my name is and I'm from Nigeria I'm an undergraduate student of the University of Nigeria where I study mechanical engineering and yeah my hobbies are listening to good music and playing stuff those that's my right here so my machine Learning Journey I actually began machine learning in 2021 uh I come from a place in the world where artificial intelligence has not really gained his ground so I did a lot of self learning I leveraged on free coures on platforms like postera to get started and I'm actually quite new to natural language processing I think a CO dive into the FI about a month ago yeah so in my project presentation I'm going to answer three questions what why and how so um what did I do I bu an app that helps students generate creative project topics based on their discipline so um I know the title say engineering students yeah those are the people I have in mind actually when building the app but it actually works for any discipline you find yourself especially technical disciplines like um computer science physics and the rest so a nice feature of the art is that it allows the user to enter his or her favorite CES in order to streamline the generated project topics to suit the interest of the user so like I said I'm studying mechanical engineering and obviously I have um topics I love more than the other this feature actually allows the user to impute the CES that interest IM to be able to streamline the generation of this topics to the interest of the user why did I build it it was needed at the time by people around me so I'm I'm in my final year and as a requirement graduate to um to get a bachelor engineering degree you actually have to do an undergraduate project so at the point um we asked to commit um project proposals to professors to actually see which project we actually have in mind to do so I was getting some complaints that by some people that they were not able to actually come up with something nice so I thought why not just do it so yeah that's one reason why I did that I wanted to create a readily available to that helps to get project topics that help students to get project topics that will excite them it's also felt exciting to solve a problem in my immediate environment so how did I build it I used and the um generate end points and EXT large model i b the front end streem and and the app was deployed on stream Cloud let me just do a quick live demo how the app actually works this is what the app looks like so um project idea generator so basically you put in your department I put in mechanical engineering then you put in your favorite courses and I put fluid mechanics and machine design then it puts the number of project topic ideas you want then you can generate project topic ideas so it will bring out the project topics and and a very few um words describing the project so you can see cfd analysis of helical gear form you can see this you can see this one 2 3 4 yeah and five so that's actually how the uh app works so um what next I plan to work on this projects uh by adding extra pictures such as the ability to generate links to Publications on similar projects as the one suggested and I also look forward to building more projects for her yeah thank you wonderful thank you so much for the demo presentation um we have a question from Serge on the chat did you fine tune the model or did you just use prompt engineering I I just use prompt engineering I didn't yeah awesome uh folks do you have any comments questions thoughts on how to improve it further yeah Toby I think this project is really cool um I uh one of the things that I would be really interested in is what you said which was getting access to other Publications that could relate to the topics being generated I think a lot of the problem that people have with literature review is just like finding the stuff that's relevant so if you get like a topic and then match it to like hey these are the other articles you should read to to learn more about this thing that would be so cool that' be really neat thank you much awesome David says it's very useful and he loves the idea to add links with related project papers if you want to follow up David go for it no I think I the his second iteration and version two sounds exactly what I think would be useful um adding in papers I think would help ground the seemingly random generate ideas with actual projects that been done before it also gives students the ability to follow up and get resources from people who have done similar types of work and not have to go do that themselves but I think this is super interesting was were the titles generated separately from the related content beneath it or was it generated as one request no no no they were generated separately yeah yeah ok does um Robin says he would love this essay idea when he was in school I think many of us would benefit from a all like this um we have a moment for like final comment or question or feedback if you want shoot all right I'm trying to understand because I already not into um the whole AI that much but I've noticed that something like open ey they have um a language model that you could type in something and get an answer to the question so I was wondering how your output compared with doing something like that in open language model I didn't really get your question someone just me to Stanley I think your question is if he's tried his prompt on uh on the open AI end points that's your question yeah I I have actually not used open air before I I'm actually new to this machine learning concept so I I don't know if I could answer your question perfectly having seen your prompts uh Stan I could probably answer that question so like open I also provides a large language model via an API so using his prompts on open a would probably get similar results both the language models are different and when like third parties have evaluated them um they they often end up roughly roughly equivalent um he might need to change his prompt a little bit in order to get the best results from open AI um but the results would would be fairly fairly similar I imagine for up promps like this okay all right than thanks much no worries righty thanks so much folks for your questions and feedback and comments um thank you Toby for your lovely presentation we are moving to the third demo of this session which is going to be presented by Amir he is uh going to show us his super Transformer which organizes emails based on urgency and importance I'm your over to you thank you so yeah thank you everyone uh welcome to the demo of super Transformer my name is Amir and we buil this prototype as part of the coer haathon classify API uh that was ConEd by lab so a bit about myself I have say about 15 years of experience in Tech and have played various roles from Individual contributor to head ofing but I am truly a ticker at heart and I like to play with the technology uh anything up and coming uh with NLP I had uh I have a bit of a history I started dabbling with NLP say about 10 years back but back then it was mostly statistical um bag of words what at um I did cor Manning jski but I had a feeling you know this is not ready for uh some sort of a industry and things uh fast forward 5 years and uh thanks to Transformers and attention mechanism and architecture uh every day we are basically getting papers that are just feating the S benchmarks and the field has got so exciting I was just looking at stable diffusion and also so mindblowing and so much potential over there so um I wanted to devel with technology and the best way to do it is is part of haathon it just forces you to you know come up with something very useful very quickly so what is super Transformer super Transformer is an AI assisted email inbox for productive efficient and effective executives super Transformer the word itself is a play on superum uh email inbox manager with similar goals of making you very efficient super Transformer allows you to intelligently categorize your emails into actionable labels so that you spend less time pring uh through your emails and more time focused on your work so uh I'll raise the stakes a bit um if you can just take a quick 10 20 seconds to send an email with a subject line of your choice uh to test. Hillary gmail.com test. Hillary gmail.com this is going to be useful later in the demo you can take a subject line one of the uh displayed over here or you can just uh you know send a subject line of your own um that you can share let me quickly check if I have any new emails uh this is the email inbox is going to bece it okay so uh during the Remo at any point if you can send out an email that will really help with them anyways jumping on to the next what is the problem that we are trying to solve and how we are trying to solve it so it's not a New Concept it's a very very old productivity hack forly known as Eisen how decision metrics so president Eisen how had a quote uh something which is urgent is not important and something which is important is not urgent uh people have like taken this particular code and try to basically create Frameworks out of it and what we have is we classify task in these two axes important and Urgent so something that is important and Urgent uh you know uh it's better if you just do it right away something that is important that not urgent you can decide later maybe in the evening end of day something which is not important but urgent needs to be responded quickly you can just delegate it to uh your colleagues or um to your one downs and something that is not important not urgent it's better to just discard it and have less uh distraction so that is basically the concept around the Isen how metrics but if you look at other productivity metrics like inbox zero or getting things done as well as a lot of other producs follows there is a lot of overlap and they basically follow some sort of a similar philosophy jumping on to uh a solution um so uh we were looking uh the team was looking uh during the haathon we were looking for email data that we can use for the Prototype most of the email data uh C to to spam not spam type of problem but then we found the Hillary Clinton email ter and uh the Clinton emails were released as part of the public investigation uh that was because of her using her private email servers and things and that was a perfect fit for our use case Clinton being the top most executive of the state department needs to be very highly efficient to be effective at her job so uh rather than just creating some sort of simulated data this was a real world data and if you can demo our use case on this real world data of how we can make Clinton efficient at her work uh is going to be you know uh just uh um the testimony of this is going to work in the real world so uh what we have used various open source Technologies as part of it we used label Studio to uh manually label the email for training uh we we used U uh cohere apis to rain and fine tune a model to classify into important verion or those four categories that I actually show to you uh use streamlet to host the public and we use f them P them is a IP so so it basically constantly checks your or basically pings your Gmail server and as soon as it receives an email it takes it through its workflow and there we figure out if this email is important or urgent and based on it we can actually use the Gmail apis to call back from py to label it in one of those four categories so enough of talk let actually let's actually see the demo nice thank you so much for everyone who sent out the email um so let's actually see uh this is P3 uh I have paused the workflow otherwise it would have you know um uh just triggered automatically so I unable the workflow I go inside I trigger it uh I trigger it uh basically uh manually but it can actually get triggered every minute every five minutes based on it and uh you see a lot of new emails have come up a lot of new tasks have started so let's actually see if basically you know starts labeling oh it has started labeling you can see a few of the emails have got start labeling here uh something which is like you know disc uh Happy Anniversary or thank you for your time that's labeled as discard something like you know uh can we request a meeting gets labeled as delegate so that your executive assistant actually can get into it and things so apologies to everybody whose email got labeled as discard this is finding on Hillary Clinton's email data so don't take it personally um going so uh going back here if you want to try it out how it works uh and if you want to just try out a few things on your own you can actually go to this particular URL uh pit. leup Transformer one word and here you can actually just uh write down the subject and the body I'm just going to give you the same results as well as more details around how the four years classify model what is the r is returning and what we are leing in so Iran China deal maybe later it's it's important but not really urgent uh maybe something like dprk that should ring bells in State Department click on it definitely you should look at any emails that uh says dprk amets about conventions obviously the state Secretary needs to actually quickly look into it so that was the demo so yeah uh if you if I have been uh at one of those points where I was getting a lot of emails in my inbox and there was no way to you know understand what to attend to and what to attend to not with super Transformer he he plan to actually you know categorize it so that it's quickly actionable you just go to your now emails reply right away later emails at the end of the day discard you don't even have to look into and delegate you can just create a filter to send forward it to your executive assistant or secretary or other things uh you are you have more time for your work and less time to the emails a few of the challenges uh the data was not clean they're using very few signals and things around it there is definitely a potential uh in it I think you uh uh we do see a lot of potential with that I'll pause my demo beautiful thank you so much um incredible idea I think many of us here would benefit from a filter powered by a large language model on our inboxes um yeah folks thoughts questions feedback do you test various sizes and amounts of dat to fine tuna classifier because I guess the follow-up question is how realistic would it be to have individually customized filters for everyone's inbox and how much data do you think that would take for every person yeah I think it's a very interesting problem I think uh it also touches upon these Federated models right where a model is useful for multiple users but then there is one model which is personalized for you so uh what we have done is we have used the real world data and if you actually look into there is also one of the constraints any urgent and important communication also happens over WhatsApp so from that particular Point like we are not really monitoring WhatsApp uh so that skewes uh our understanding of it uh we have used the real data so whatever was in the email body uh we basically definitely used it uh but then the other thing is we are discarding a lot of signals like where it is coming from who's sending it at what time of the day are sending and uh yeah uh also the threaded conversation uh it was like too complicated for hackathon to to like go into so a lot of those signals we just discarded just keep just focus on awesome Nick has a question um that he posted in the chat do you want to share it directly Nick oh yeah just you were classifying uh you were classifying based off of just the subject line the demo so do you like are you normally using the subject line and the body or just the subject and then if you normally use both does just giving it a subject like affect the the accuracy or the performance yeah actually we are using both so we are just concatenating the subject and the body with some sort of a separator and sending it for find uning so both the subject as well as the body but you find that it's like robust it's robust to have only the subject like it's able to classify still accurately with just that uh yeah I think even if you look into the the actual email d a lot of communication just happens to the subject line a lot of times the body is so from that perspective also I think yeah definitely most stress on the subject but I think yeah uh I'm still like confused how am I going to tell the model that look focus on the subject more and body less should I just repeat it more multiple times it's going to interes cool thank you thank you awesome we have uh space for one more questions feedback comments so go ahead and grab it otherwise we will move to another demo harada has a question does it automatically categorize emails as important if subject CL has the word important um not really I would say like important is a very generic word um so we have manually categorized the emails based on our own geopolitical understanding so if we found the actual email dump having uh dprk or liia or other keywords we classify it as important analoging um and yeah so so that's how the model is strained so it's not really going to on to the word important but I think uh there is a lot of correlation also like if something needs your attention immediately the subject line automatically contains word urgent so if you send the email with subject line urgent and maybe not so urgent think uh the model might be confus and classify it as important okay har says it's very useful uh and thank you so much am here for your lovely demo we loved it um let's move on on to our final demo of tonight which will be presented by Arnav Atara and Sati hopefully haven't butchered the name completely if so please forgive me um so folks will be showing us scripture um which is fantastic creature generator for writers bring it on so welcome to our project demonstration for scripture it's a text and image gener software for advanced script writing and creativity um so we can move on to the about ourselves next slide so this is our team um sa can start I'm sa I'm a freshman at Berkeley and I'm majoring in business as well as Premed and I'm Arnav I'm a computer science student at UCSC also a freshman and my experience with natural language processing is mostly through um research and explainable artificial intelligence I I'm a freshman in Berkeley majoring in computer science and econ or I guess um so we're going to start with a demo and then go into the inner workings of the project the demo will highlight a use case and then the functionality so um if you can stop sharing the screen and I'll share mine so let's start with a demonstration that will cover both the use case and functionality um let's say I'm a writer who is writing a traditional Quest story and I know that my hero is going to find himself in a cave and and I know that he's going to encounter some sort of creature but I can't think of the motivation for an entire creature its description and its backstory so I launch the scripture tool and type in something as simple as cave monster submit and um after it's running a few models behind the scenes it's going to Output a both an image for this monster and a description and so here we go it's a as you can see fairly realistic image of a monster and then a rather good demonstr a rather good backstory for the monster so it says it comes from a mysterious island it's a little brown animal whose name can be translated to small cold thing and it's a male humanoid and it has a helmet with horns so just like that with a click of a button I have unlocked an entire section of my entire my book I now have a description for my creature which I can write about I have a rough backstory for him and now I can launch my writing based on this story and the background I can generate an infinite amount of characters and infinite amount of text and it works as for as much as I want and it completely offloads the creative inspiration I need so let's that was it for the demo let's go into the inner workings all right so scripture uses a multimodal approach um which starts with coheres version of stable diffusion it's it's as simple as importing a prompt and then um getting a text output image output we then take this image output and run it through an image captioning model called blip and finally we take that text from the caption model and run it on a fine-tune generative model which a there will talk about later which gives us our overall output for the backstory we wanted this project not just to focus on generating a rough description of the character but also a story for the character as that's really important for writers so we managed to do that with fine-tuning to start off um we'll just talk about the implementation for stable diffusion which was really simple thanks to the coher API it's a simple as a Json request and as you can see it was three Lin of code originally we were going to use Google dream booth for f tuning stable diffusion on data sets that we found and the results were very good but we found it was pretty hard to run locally and so we ended up with this and as you can see by the demo the results are very promising we then implemented blip which is an image captioning and visual question answering model which works very accurately as you can see by the um screenshot of demo that I took uh it's very accurate for realistic images for Fantasy images it's not as great but um it is still really good for our use case and then other will'll go on to talk about the generative part of our application yeah so now we this is where we started implementing the text generation portion of the project and for this we use used coher API um text generation functionality and we imp we inputed the caption generated by blip to generate like a background story uh for each of the for each of the images so to ensure that the generation that the text generation was customized towards fantasy we had to find um a lot of like data about character backgrounds and we we stumbled out upon a website that provided exactly this it was a Wiki of uh mythical creatures and it had over 250 such descriptions so we used um beautiful soup to scrape U the descriptions and format it so that it could be taken in by um the coher API and we used this data to um train the model and this and all this produced our final project which was shown in the live demo so next I'll hand it to C A to talk about um use cases thank you so some use cases for scripture not only is it fun to play around with but it has a lot of applications in mainstream entertainment with creative writers and creative teams so it can be used from anything from film to video games to books and we even thought about card games like I don't know if you guys played Yu-Gi-Oh as a kid but I was big into it um scripture makes static worlds more vibrant and Rich writers have a access to a large data set where they can draw upon a variety of characters and a variety of Back stories to help with their World building and story development not only that but scripture reduces creative load on writers and developers recently it's really hard for um writers and developers there's large teams in companies especially such as Marvel to um create such characters and backstories for cre uh such characters so it would reduce the cost of um creative teams and it would increase the efficiency at which they're able to create said characters and some future development for scripture we thought about diverging just from fantasy characters to a wide genre of characters um this would like create a larger market and would also create a larger audience for scripture um not only that we also thought about generating Sprites with backstories this would be 3D meshes which could be used in video games as well as VFX and this would have a larger application in um film and uh TV shows and finally metaverse which is an expanding area of tech um has not so populated worlds right now so we thought that we could use scripture and the idea of creating 3D meshes to um populate world and create a larger user engagement and this would also um help streamline game and World development with uh creators being able to create more Sprites and more characters for their set video games or worlds and with that I think that's the conclusion of our presentation thank you a so much thank you France that was awesome that like there is so much love you got in the comments during your um I just want to highlight that um there is one request for a prompt maybe you want to run it meanwhile while you're answering questions and responding to feedback nick uh would you like to voice out your idea oh yeah yeah my question is this is like this is like two steps away from being super useful for Dungeons and Dragons have you guys ever played Dungeons and Dragons yeah like if you just conditioned off of um yeah if you just conditioned off of like a level like a level and a and an enemy type or something it would be really great like suddenly every single NPC could have a could have a a full backstory so I wonder if you guys looked into that and then the other question I had is I noticed that in your your character description there was Vivid like um visual information like you say he's got a horn a helmet with horns are you putting that into the into the art generation prompt or are you generating and then it ignored it or are you generating those two things independently all right so to your first question about DND that was actually the inspiration for this project uh a lot of time was spent trying to find like a DND data set um for both the monster generation and the U text generation and then um so your second question was about whether the prompt or whether the text generation uh included The Prompt or was it like yeah is one of the art generation like you generate text describing the creature do you use that text for generating the image or you just using like the name yeah so we just used the original prompt and so that was as simple as like cave monster and then it implemented that then blip uh ran an image captioning on it and um outputed a result and then so that caption was taken by the text and um generated from there wait the caption was generated from the image yes uh okay okay okay because yeah it said a few things that weren't in the image like it described a helmet with horns that the guy wasn't wearing but I wonder if you did it in the opposite way like first use coher to take a cave monster and then generate a visual description of the cave monster and then put that into the image generation as opposed to it sounds like you guys are doing the opposite you want cave monster image text description you might go cave monster text descrip description image and get and get more cohesive results right yeah um there's like there's also a lot of fine tun we could do with the blip model um it was actually just a simple pre-trained implementation but yeah changing the order of the pipeline could also increase its accuracy even I'm not sure though but that's cool Keith is asking whether you can try the link to your project and asking yeah I just need to um it's being run locally so I just need to use radio to share it awesome once you do that we will share with uh everyone the link and um we are running short on time so I will have to uh put a pause on our discussion we can move it to community demos Forum uh where we will be posting also recording from this session meanwhile I want us to move to the thank you so thank you so much first of all um folks for presenting your demo we are coming to um the segment in which Aiden is choosing his favorite demo and I actually saw Aiden here during the event so drum rolls oh actually a little comment first before you Aiden share you will share your opinion I just want to let you know that Aidan will pick his favorite demo from this event it doesn't mean that other demos weren't good we love all of them that's why you're here that's why you're presenting today and talking with us about it but we just hope that it will extra motivate you and will be like this nice chair on top the folk that will or folks that will get picked by Aiden today will have a one-onone mentoring session with him bya Discord and we'll get a Discord badge uh affected the 1 of November so it will be reflected there and will be featured on our socials so we will push a little bit of promotional love into that particular demo um we are also going to add uh this demo to the conference uh Booth uh during one of our upcoming conference appearances so yeah putting that lovely demo out there and without further Ado um Aidan do you want to share your favorite pick I uh think I didn't realize we had uh such a great graphic design uh that's me app um yeah so amazing amazing stuff it was so cool to to see what you're building um all four of the projects were were super sick I uh as soon as they are sharable please let me know because I'd love to play with them and like promote them on on Twitter um but I have to pick one which sucks for me um and I think I think I'm going to have to go with the one that I feel like I need the most right now which is the super transformed my inbox is crazy and uh I really want to install this um so tell me how to do it I uh give me a give me a link I'll install it um but thank you all so much I'm so stoked by all the projects they were amazing and I'm I'm stoked to meet you Amir congrats to Amir super Transformer you're the one thanks Aiden for um chiming in well uh everyone it was such a fantastic lineup today thank you so much for coming thank you so much for your presentations thank you folks for for participating in our collab we have planned a little group photo if you're up for it so um if everyone could turn on their cameras and make a cute face for the photo so that we can capture you all um that would be great let's take a photo for in three two one all right that was the cutest I could do at this time of the day thank you thank you all so much let's wrap up I want to give you uh just a little bit of a hint of what's coming uh up for uh the future events so then the next collab will uh be held at the last Friday of November and then the final one this year will be held mid December 16th of December if you haven't signed up yet make sure to sign up if you want to present your demo next time make sure to let us know uh we are also having quite some onboarding webinars so if you're curious about using coh here if you haven't had a chance to play around with it you want to know the basic ins and outs of our API and playground please check them out and we are also having a next episode of talking language AI series with our lovely Jay alar who is also present today coming up soon so stay tuned for that if you uh are going to scan this little cute QR code it will take you to our Discord this our Discord invite we have a community demos Forum where you can share your projects share your opinions about the event ask questions ask followup questions uh and keep up to date with what's going on in our community all right wishing everyone a fantastic rest of the day or night depending on where you are thank you so much for coming it's been great and see you next time take care thank you thank you Sandra thank youen bye everybody great job bye everybody bye [Music]