Learn to Draw Real People using AI: Unveiling Future of Image-to-Image Translation

Key Takeaways

you can now generate high-quality face images from ruff or even incomplete sketches with zero drawing skills using this image to image translation technique if your drawing skills are as bad as mine you can even adjust how much the eyes mouth and nose will affect the final image let's see if it really works and how they did it recent deep image to image translation techniques allo fast generation of face images from freehand sketches however existing solutions tend to overfit two sketches thus requiring professional consistent sketches as inputs show you Chen and I'll just share the paper called deep face drawing deep generation of face image from sketches to address this issue their key ID is to implicitly model the shape space of plausible face images and synthesize a face image in the space to approximate an input sketch not worry if you are lost we will cover this in the video there system allows user with little or no training in drawing to produce high-quality face images from ruff are even incomplete freehand sketches the method even faithfully respects user intentions in input strokes which serve more like soft constraints to get image synthesis they essentially use input sketches as soft constraints and is thus able to produce high-quality face images even from these rough sketches most of such deep learning based solutions for sketch to image translation often take input sketches almost fixed and attempt to infer the missing texture or shading information between strokes to some extent their problems are formulated more like reconstruction problems with input sketches as hard constraints since the often trained their networks from pairs of real images and their corresponding edge maps due to the data-driven nature the you cry or test sketches with quality similar to the edge maps of real images to synthesize realistic face images however such sketches are difficult to make especially for users with little training enjoying to address this issue the key ID is two implicitly learn a space of plausible face catches from real face catcher images and find the closest point in the space to approximate an input sketch this way sketches can be used more like stuff constraints to guide the image synthesis and as you can see the results are amazing as illustrated the deep learning framework takes a sketch image as input and generates a high quality facial image the network's architecture consists of two sub networks the first sub network is the component embedding module which is responsible for learning feature and beddings of individual face components using separate auto encoder networks the step turns component sketches into semantically meaningful feature vectors using a non - encoder architecture that separately learns five feature descriptors from the face cache data namely for left eye right eye nose mouth and reminder a reminder image corresponding to the reminder component is the same as the original sketch image but with the eyes nose and mouth removed the second sub Network consists of two sub modules the feature mapping and the ml synthesis although FM looks similar to the decoding part of see using the coding models converting feature vectors to spatial feature Maps it improves the information flow and thus provide more flexibility to fuse individual phase components for higher quality synthesis results the feature maps of individual phase components are then combined according to the face structure and finally pass to the is for face image synthesis which converts them to a realistic face image using a conditional Gann architecture which takes the feature Maps as input to a generator with the generation guided by a discriminator if you are not familiar with the gun architecture I suggest you to watch the video I made introducing them with this complex architecture they adapted a two-stage training strategy to train their network in stage 1 only the C module is trained by using component sketches to train five individual auto-encoders for filter embeddings the training is done in a self supervised manner which are covered in a previous video and I linked in stage 2 they fix the parameters at the Train component encoders and train the entire network with the unknown parameters in the FM and is modules together in an end-to-end manner to assess users especially those with little training and drawing we provided a shadow guidance caching which is shown in this video given a current sketch it finds the 10 most similar sketch component images the fun component images are then blended as shadow and placed at the corresponding components position for sketching guidance as you can see on the left initially when the canvas is empty the shadow is more blurry the shadow is updated instantly for every new input stroke the synthesized image is displayed in the window on the right users may choose to update the synthesized image instantly or trigger a convert command of course users with good drawing skills tend to trust their own drawings more dentals with little training enjoying so they provided a slider for each component type to control the blending weights between a sketch component and its refined version controlling the degree of interpolation between the sketch you made and the final version that is shown for either the eyes nose or mouth both qualitative and quantitative evaluations show the superior generation ability of their system to existing and alternative solutions just take a moment to look at these amazing results in comparison with the alternatives creating realistic human face images from scratch benefits various applications including face morphing face copy/paste criminal investigation character design it's casual training and more due to their simplicity conciseness and ease of use sketches are often used to depict desired faces which makes this new paper extremely relevant of course this was just a simple overview of the new image to image translation technique that allows fast generation of face images from freehand sketches I strongly recommend to read their paper and check their video demo both are linked in the description if you want me to cover a specific term in next video leave it in the comments below please leave a like if you learn something and subscribe to the channel to not miss any further terms clearly explain [Music]