Transformers Can Do Arithmetic with the Right Embeddings #ai #machinelearning #engineering

Key Takeaways

hello everyone very excited to feature this new paper that achieves 99% accuracy on 100 digit addition problems by training on only 20 digigit numbers with a single GPU the main challenge this work addresses is the inability of Transformers to track the exact position of digits they do this by adding an embedding to each digit that includes its position relative to the start of the number these gains also transfer to multi-step reasoning tasks that include sorting and multiplication it's quite interesting to see such a generalization to un digit lens with such a simple modification this is an indicator that Transformer architectures have the potential to benefit from a variety of modifications that improve generalization or unlock other capabilities leave a like and follow for more of these paper summaries