GPT-OSS: The New AI Model Explained (Context Window!) #shorts

Key Takeaways

GPT OSS is one of OpenAI's most anticipated recent launches. A large fully open weights model from one of the leading American AI labs. Let's take a closer look at the paper to find out how it was actually engineered and trained. GPT OSS is a mixture of experts model available in two sizes, 120 billion parameters and 20 billion parameters. Each token activates the top four experts, meaning only a portion of the total parameters are used at any given time. This allows for efficient inference without sacrificing the benefits of a larger model. Trained as a decoderonly transformer, GPTOSS incorporates plenty of features typical to modern LLMs. This includes grouped query attention, a modified attention mechanism that lets multiple query heads share the same key value pairs to reduce memory use and speed up inference. It also includes swiggloo activations in the feed forward network layers which allow for more nuance transformations than simpler activations like relu as well as rotary positional embeddings or rope which encode token position directly into the attention mechanism to support longer contexts. Finally, the model also makes use of RMS norm with pre-normalization, a normalization method that scales inputs by their root mean square for more stable training. One standout capability of the model is its 131,000 token context window, which it achieves by applying yarn scaling during pre-training rather than as an inference time adjustment. We'll touch on what this means a little bit later in the video. For GPTO OSS, OpenAI makes use of their open- source 0200K Harmony tokenizer.