Agentic Topic Modeling with Maarten Grootendorst - Weaviate Podcast #126!

Maarten Grootendorst is a psychologist turned AI engineer who has created BERTopic and authored "Hands-On Large Language Models" with Jay Alammar. The rise of LLMs and Agents are transforming many areas of software! This podcast dives deep into their impact on Topic Modeling! Maarten designed BERTopic from the start with modularity in mind -- letting you ablate embedding models, dimensionality reduction, clustering algorithms, and more. This early insight to prioritize modularity makes BERTopic incredibly well structured to become more "Agentic". An "Agentic" Topic Modeling algorithm can use LLMs to generate topics or topic descriptions, as well as contrast them with other topics. It can decide which topics to subdivide, and it can integrate human feedback and evaluate topics in novel ways... I hope you find the podcast interesting! Links: Hands-On Large Language Models: https://www.oreilly.com/library/view/hands-on-large-language/9781098150952/ BERTopic: https://github.com/MaartenGr/BERTopic BERTopic (paper): https://arxiv.org/abs/2203.05794 Learn more about Maarten Grootendorst: https://www.maartengrootendorst.com/ TopicGPT: https://arxiv.org/abs/2311.01449 TnT-LLM: https://arxiv.org/abs/2403.12173 Chapters 0:00 Welcome Maarten! 1:57 Hands-On Large Language Models 7:34 An Overview of Topic Modeling 10:45 LLM Topic Generation 17:13 Topic Modeling with Human Feedback 21:33 Topic Granularity 26:00 Visualizing Topics 31:18 Contrastive Topics 33:24 LLM-as-Judge for Topics 39:06 Separating Generation from Assignment 44:20 Applications of Topic Modeling 55:14 Semi-Supervised BERTopic 1:01:06 Future Directions for AI