Leviathan: Decoupling Input and Output Representations in Language Models
📰 ArXiv cs.AI
Learn how Leviathan decouples input and output representations in language models to improve token representation and vocabulary discrimination
Action Steps
- Implement learned embedding vectorization (LEV) in a Transformer architecture to replace traditional input embedding matrices
- Decouple input and output representations to separate token representation and vocabulary discrimination objectives
- Use Leviathan's output head to project embeddings onto a vocabulary
- Evaluate the performance of Leviathan on benchmark tasks to compare with traditional language models
- Apply Leviathan to downstream NLP tasks to leverage its improved token representation capabilities
Who Needs to Know This
NLP engineers and researchers can benefit from this approach to improve language model performance and efficiency
Key Insight
💡 Decoupling input and output representations in language models can improve token representation and vocabulary discrimination
Share This
🚀 Introducing Leviathan: a new Transformer architecture that decouples input and output representations for improved language modeling #NLP #LLMs
Key Takeaways
Learn how Leviathan decouples input and output representations in language models to improve token representation and vocabulary discrimination
Full Article
Title: Leviathan: Decoupling Input and Output Representations in Language Models
Abstract:
arXiv:2601.22040v2 Announce Type: replace-cross Abstract: Modern language models use a single matrix for input embedding and output projection. This couples two distinct objectives: token representation and discrimination over a vocabulary. This work introduces Leviathan, a Transformer architecture that replaces the input embedding matrix with learned embedding vectorization (LEV), a compact continuous mapping from token indices to embeddings. Leviathan's output head remains untied for a paramet
Abstract:
arXiv:2601.22040v2 Announce Type: replace-cross Abstract: Modern language models use a single matrix for input embedding and output projection. This couples two distinct objectives: token representation and discrimination over a vocabulary. This work introduces Leviathan, a Transformer architecture that replaces the input embedding matrix with learned embedding vectorization (LEV), a compact continuous mapping from token indices to embeddings. Leviathan's output head remains untied for a paramet
DeepCamp AI