Gradient Descent from First Principles: Why Adam Outperforms SGD on Transformers

📰 Medium · Machine Learning

Learn why Adam outperforms SGD on transformers through a first-principles analysis of gradient descent

advanced Published 29 Apr 2026

Action Steps

Apply gradient descent from first principles to understand optimizer behavior
Compare the performance of Adam and SGD on transformer models
Analyze the impact of adaptive learning rates on model convergence
Implement Adam optimizer in your model training pipeline to improve performance
Evaluate the trade-offs between Adam and SGD in terms of computational cost and accuracy

Who Needs to Know This

Machine learning engineers and researchers can benefit from understanding the differences between Adam and SGD optimizers to improve their model training workflows

Key Insight

💡 Adam's adaptive learning rate and momentum terms allow it to outperform SGD on complex models like transformers

Key Takeaways

Learn why Adam outperforms SGD on transformers through a first-principles analysis of gradient descent

Full Article

Title: Gradient Descent from First Principles: Why Adam Outperforms SGD on Transformers

URL Source: https://levelup.gitconnected.com/gradient-descent-from-first-principles-why-adam-outperforms-sgd-on-transformers-4f88ec6ecf3e?source=rss------machine_learning-5

Published Time: 2026-04-29T14:51:30Z

Markdown Content:
# Gradient Descent from First Principles: Why Adam Outperforms SGD on Transformers | by Armin Norouzi, Ph.D | Apr, 2026 | Level Up Coding

[Sitemap](https://levelup.gitconnected.com/sitemap/sitemap.xml)

[Open in app](https://play.google.com/store/apps/details?id=com.medium.reader&referrer=utm_source%3DmobileNavBar&source=post_page---top_nav_layout_nav-----------------------------------------)

Sign up

[Sign in](https://medium.com/m/signin?operation=login&redirect=https%3A%2F%2Flevelup.gitconnected.com%2Fgradient-descent-from-first-principles-why-adam-outperforms-sgd-on-transformers-4f88ec6ecf3e&source=post_page---top_nav_layout_nav-----------------------global_nav------------------)

[](https://medium.com/?source=post_page---top_nav_layout_nav-----------------------------------------)

Get app

[Write](https://medium.com/m/signin?operation=register&redirect=https%3A%2F%2Fmedium.com%2Fnew-story&source=---top_nav_layout_nav-----------------------new_post_topnav------------------)

[Search](https://medium.com/search?source=post_page---top_nav_layout_nav-----------------------------------------)

Sign up

[Sign in](https://medium.com/m/signin?operation=login&redirect=https%3A%2F%2Flevelup.gitconnected.com%2Fgradient-descent-from-first-principles-why-adam-outperforms-sgd-on-transformers-4f88ec6ecf3e&source=post_page---top_nav_layout_nav-----------------------global_nav------------------)

![Image 1](https://miro.medium.com/v2/resize:fill:64:64/1*dmbNkD5D-u45r44go_cf0g.png)

## [Level Up Coding](https://levelup.gitconnected.com/?source=post_page---publication_nav-5517fd7b58a6-4f88ec6ecf3e---------------------------------------)

·

[![Image 2: Level Up Coding](https://miro.medium.com/v2/resize:fill:76:76/1*5D9oYBd58pyjMkV_5-zXXQ.jpeg)](https://levelup.gitconnected.com/?source=post_page---post_publication_sidebar-5517fd7b58a6-4f88ec6ecf3e---------------------------------------)
Coding tutorials and news. The developer homepage [gitconnected.com](http://gitconnected.com/)&&[skilled.dev](http://skilled.dev/)&&[levelup.dev](http://levelup.dev/)

Member-only story

# Gradient Descent from First Principles: Why Adam Outperforms SGD on Transformers

[![Image 3: Armin Norouzi, Ph.D](https://miro.medium.com/v2/resize:fill:64:64/1*TsqarG7f8uUnyvK9Dd2wcg.jpeg)](https://arminnorouzi.medium.com/?source=post_page---byline--4f88ec6ecf3e---------------------------------------)

[Armin Norouzi, Ph.D](https://arminnorouzi.medium.com/?source=post_page---byline--4f88ec6ecf3e---------------------------------------)

10 min read

·

Just now

[](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F_%2Fvote%2Fgitconnected%2F4f88ec6ecf3e&operation=register&redirect=https%3A%2F%2Flevelup.gitconnected.com%2Fgradient-descent-from-first-principles-why-adam-outperforms-sgd-on-transformers-4f88ec6ecf3e&user=Armin+Norouzi%2C+Ph.D&userId=7d14ec5b901a&source=---header_actions--4f88ec6ecf3e---------------------clap_footer------------------)

--

[](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2F_%2Fbookmark%2Fp%2F4f88ec6ecf3e&operation=register&redirect=https%3A%2F%2Flevelup.gitconnected.com%2Fgradient-descent-from-first-principles-why-adam-outperforms-sgd-on-transformers-4f88ec6ecf3e&source=---header_actions--4f88ec6ecf3e---------------------bookmark_footer------------------)

[Listen](https://medium.com/m/signin?actionUrl=https%3A%2F%2Fmedium.com%2Fplans%3Fdimension%3Dpost_audio_button%26postId%3D4f88ec6ecf3e&operation=register&redirect=https%3A%2F%2Flevelup.gitconnected.com%2Fgradient-descent-from-first-principles-why-adam-outperforms-sgd-on-transformers-4f88ec6ecf3e&source=---header_actions--4f88ec6ecf3e--------------

Read full article → ← Back to Reads

Gradient Descent from First Principles: Why Adam Outperforms SGD on Transformers

Key Takeaways

Full Article

Related Videos