Lightning Talk: HieroGlyph2Text: A PyTorch-Powered Pipeline for Automated Egyptian H... Susi Gentsch

Key Takeaways

thanks everybody for coming today I will present to you my project my bachelor thesis about I will present to you how I set up a pipeline to translate Egyptian hieroglyphs from an image first a bit about myself my name is suan Gen and I'm currently finishing up my bachelor degree at the University of Bon studying computer science with a focus on computer vision the project that I'm presenting today to you is done in cooperation with Bond train a program of my school where students and AI experts come together with subject matter experts to find solution for their problems here's a quick overview about my pipe about heroglyph to text at first the user inputs a image with a prompt with object detection the pipeline then cuts out the single hers and passes them into the classification model which returns Garner codes these par codes are then translated with a language model using Rag and out comes the English an attempted English translation I want to go give you a quick overview about what I'm talking today at first I cut down the problem then I will present to you the data sets that I used to train the models followed by going in depth about the heroglyph detection the classification that I have done with pytorch and a translation that's done using rag with Lama 3.1 in the end I will give you a quick demonstration about how it works first when we want to translate heroglyph from an image this problem can be cut down into three sub problems first the detection of the hieroglyphs then their classification followed by the translation since you can't write hieroglyphs with your keyboard egyptologist use Gardener codes as a lookup table for them as we all learn in elementary school tool I'm sure we know that this small image of a bird is called g17 a few challenges when it comes to dealing with hieroglyphs are first of all there are a lot of them hundreds of them and when we see them in the field they're often not perfect they're malformed are an irregular grouping and when it comes to the translation the same group of heroglyph doesn't always mean the same thing have the same translation the translation often depends on and the broader context of where the hieroglyphs appear the data sets that I used for my model training are first of all the Frank Morris data set for the image detection and classification this is a data set with images with 10 images of the walls of the pyramids of unas along with them are about 4,000 annotated image of images of individual hieroglyphs along 162 class of different Gardener codes for the language part I created a custom rack data set that contains about 1,000 lines entries saved in adjacent lines format that and each entry contains a unique Gardener code its description and or and or ideogram this is what these data set of the small images look like and here are a few example lines of the r data set now to the object detection for the object detection I decided to go with yolo V8 because it performs very well at detecting small images on large small objects on large images I trained to detect various types of hieroglyphs in an image the data set I used for that I split 70% into training 20% to validation and 10% for test now to the classification part I designed my classification model with pie torch the key points why I use pouch for this was first of all pyou has a rich ecosystem of pre-trained models like rnet and libraries that help to work with it then secondly its Dynamic computational graph was very helpful to adapt P to adapt rest net to my specific needs the py torch data loader further made it very easy to load and Shuffle the data around during the training another benefit of py toch was the seamless GPU integration when I get the when the object detection Returns the cutout images of heroglyph they're not in the perfect in a perfect square shape what reset needs to work with them so at first I standardized the image size and then I increased the and then I increased my data set by doing some standard augmentations with t vision. transform the final classification model that I ended up using was the result of extensive experimentation with different architectures and configurations of resnet the best performing model was a customized version of reset 50 in which I adapt in which the key modifications that I applied were first that I changed the input layer to only accept grayscale images and secondly I changed the final layer to be adjusted for the custom amount of classes of gardener codes appearing in the data set I split the data for the training 70% for training 70% for training 15% for validation and 15% for test the accuracy on the test data ended up being 94% which exceeds previously published results now to the final part of my pipeline the translation when I started with this part of the project Lama 3 just came out and it was not able to translate a Garder codes this is why I had to teach it what Gardner codes meant I tried to put this in a prompt input but the context window of llama 3 three was too small at the time this is why I created a my wreck my R is can be cut down into four key Parts first the indexing the retrieval augmentation and generation for the indexing I created embeddings of the rec data set with llama 3.1 these I saved in a locally in a local chroma DB Vector database once I have the database I can use it the Garner codes from the image models are being passed to the retrieval which generates embeddings for each Garner code and retrieves the matching information in the chroma DB data set base the retriev data is being saved in a dictionary containing the Gardner code description and ideogram this information is then being passed to the augmentation stage where I constructed a detailed prompt containing the received dictionary and the trans and a detailed translation instruction that emphasizes to that tells the model to emphasize on the meaning of the heroglyph rather than linguistic accuracy this prompt is then being passed into Lama 3.1 and it returns a translation here's an example of an image of the pyramids of unas I have a I passed it into my heroglyph to to text pip line and I had the first 20 hieroglyphs being translated at first the language at first the heroglyph to text Returns the Garder each Garner code with their description with their meaning and then it tries to form sentences of them of with all Garner codes combined to put it all together I created an image to text pipeline to translate Egyptian her gloves this combines computer vision with Lang lar language models in the end I used YOLO V8 for the heroglyph detection and a customized reset model for the classification the translation then happens with Lama 3.1 using my Rec system this brings an interpretive translation of hieroglyphic imagery this project lays the groundwork for egyptologists to share their knowledge and with a ease of TR talking to a language mod large language model and with their knowledge we can achieve automated real legitimate translation of Egyptian hieroglyphs I appreciate your interest in this topic and I'm open to questions thank you