Orchestrating Spark Jobs with Kubeflow // Sadik Bakiu // MLOps Meetup #71

MLOps community meetup #71! Last Wednesday, we talked to Sadik Bakiu, Freelance ML Engineer. //Abstract Apache Spark and Kubernetes have been established as de facto standards for data processing and container orchestration respectively. This talk will cover how these technologies can be integrated under the orchestration of Kubeflow. (Kubeflow has been emerging as a platform to make ML workflows easy to work with and deploy.) All codes are available here: https://github.com/sbakiu/kubeflow-spark/blob/main/kubeflow_pipeline.py //Bio Sadik is a Freelance ML Engineer focused on creating production-grade ML workflows. Since the early beginning of his career, more than a decade ago, he was fascinated by Data and Information management systems and has been working in this field ever since. Sadik also writes occasionally about technology topics. // Takeaways Understanding Spark on Kubernetes Understanding Kubeflow and Kubeflow Pipelines and its Components Integrating Spark-Operator with KFP // Other Links Medium: https://sbakiu.medium.com/ https://www.kubeflow.org/ https://github.com/GoogleCloudPlatform/spark-on-k8s-operator Contact: sadik@data-max.io --------------- ✌️Connect With Us ✌️ ------------- Join our slack community: https://go.mlops.community/slack Follow us on Twitter: @mlopscommunity Sign up for the next meetup: https://go.mlops.community/register Catch all episodes, Feature Store, Machine Learning Monitoring and Blogs: https://mlops.community/ Connect with Demetrios on LinkedIn: https://www.linkedin.com/in/dpbrinkm/ Connect with Sadik on LinkedIn: https://www.linkedin.com/in/sadik-bakiu/ Timestamps: [00:00] Introduction to Sadik Bakiu [04:40] Orchestrating Spark Jobs with Kubeflow [05:00] Agenda [05:32] Why are we doing this? 3 Stages: 1. Preprocessing 2. Training 3. Serving [07:25] We need distribution, orchestration [07:50] "In order to make the processing efficient, buying the next bigger machin