How Apache Airflow Better Manages Machine Learning Pipelines
Skills:
AI Systems Design85%
In this episode of The New Stack Makers, a trio of technologists, who all work with Amazon Web Services Managed Service for Airflow team, talked about improving the Apache Airflow user experience.
Apache Airflow is an open-source platform for building machine learning pipelines. It allows users to author, schedule, and monitor workflows, making it well-suited for tasks such as data management, model training, and deployment. In a discussion on The New Stack Makers, three technologists from Amazon Web Services (AWS) highlighted the improvements and ease of use in Apache Airflow.
Dennis Ferruzzi, a software developer at AWS, is working on updating Airflow's logging and metrics backend to the OpenTelemetry standard. This update will provide more granular metrics and better visibility into Airflow environments. Niko Oliveria, a senior software development engineer at AWS, focuses on reviewing and merging pull requests as a committer/maintainer for Apache Airflow. He has worked on making Airflow a more pluggable architecture through the implementation of AIP-51.
Raphaël Vandon, also a senior software engineer at AWS, is contributing to performance improvements and leveraging async capabilities in AWS Operators, which enable seamless interactions with AWS. The simplicity of Airflow is attributed to its Python base and the operator ecosystem contributed by companies like AWS, Google, and Databricks. Operators are like building blocks, each designed for a specific task, and can be chained together to create workflows across different cloud providers.
The latest version, Airflow 2.6, introduces sensors that wait for specific events and notifiers that act based on workflow success or failure. These additions aim to simplify the user experience. Overall, the growing community of contributors continues to enhance Apache Airflow, making it a popular choice for building machine learning pipelines.
Check out the full article on The New Stack:
https://thenewstack.io/how-apache
Watch on YouTube ↗
(saves to browser)
Sign in to unlock AI tutor explanation · ⚡30
Playlist
Uploads from The New Stack · The New Stack · 0 of 60
← Previous
Next →
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
What's Next for the Cloud Foundry Foundation in 2017 with Executive Director Abby Kearns
The New Stack
How Unikernels Can Better Defend against DDoS Attacks
The New Stack
Weaveworks is Bringing Horizontal Scaling to Prometheus
The New Stack
TNS Analysts Thanksgiving Special: The Evolution of Kubernetes and the Container Ecosystem
The New Stack
How Rancher Labs is Seeing Kubernetes Put to Work in Production
The New Stack
SAP Tests Kubernetes for Cloud-Native Enterprise Software Deployments
The New Stack
Event Marketing for Today's Developer Evangelists and Community Managers
The New Stack
NodeSource Introduces Certified Modules to Improve Node.js Security
The New Stack
How Lightstep is Illuminating the Case for Distributed Tracing
The New Stack
How OpenStack Aims to be More Inclusive without being Exclusive
The New Stack
How Shuttlecloud Saves Time and Money by Monitoring with Prometheus
The New Stack
Creating Analytics-Driven Solutions for Operational Visibility
The New Stack
Understanding the Application Pattern for Effective Monitoring
The New Stack
Building On Docker's Native Monitoring Functionality
The New Stack
The Importance of Having Visibility Into Containers
The New Stack
How Getting Your Project in the CNCF Just Got Easier
The New Stack
Tectonic Summit Pancake Breakfast: How to Sell Kubernetes to the Hypervisor-Minded
The New Stack
The Buzz at Tectonic Summit 2016 in New York City
The New Stack
Bringing Clarity to the Future of Node.js Modules
The New Stack
How FluentD Can Help Monitor Microservice Architectures Through Unified Logging
The New Stack
Reshaping Front End Development with Warehouse.ai
The New Stack
2016 Year End Wrap-Up: Discussing Docker, OpenStack, and Open Source
The New Stack
Here's Why You Should Build a Robot Using Node.JS: Because You Can
The New Stack
How the Node.js Foundation is Utilizing Participatory Governance Models
The New Stack
Set Up an MongoDB Replica Set in Less Than an Hour Using Bitnami Packages
The New Stack
Determining Who Bears the Burden of Ensuring NPM Module Security
The New Stack
How Intel Snap uses Telemetry and Kubernetes to Drive Enterprise Efficiency
The New Stack
How the NFL Scored a Touchdown with its Open Source React Framework Wildcat
The New Stack
Aporeto CEO Dimitri Stiliadis: When it Comes to Security, Context is King
The New Stack
The Buzz at Node.JS Interactive
The New Stack
Why Going Serverless Doesn't Mean 'No Ops'
The New Stack
How Node.js is Transforming Today's Enterprises
The New Stack
JJ Asghar Interview
The New Stack
How Capital One is Using APIs to Streamline Auto Financing
The New Stack
SXSW 2017: How Machine Learning Differs From Regular Programming
The New Stack
SXSW 2017: Data-Driven Applications with Capital One DevExchange's Hydrograph
The New Stack
SXSW 2017: How Good Engineers Make Bad Business Decisions
The New Stack
CloudNativeCon & KubeCon EU Pancake Breakfast 2017: Kubernetes and the Multi-Cloud
The New Stack
CNCF Executive Director Dan Kohn: What's Next for CNCF in 2017
The New Stack
Exploring the Latest Container Runtime Projects in the CNCF
The New Stack
Exploring the Future of the Kubernetes Ecosystem
The New Stack
Kubernetes and Continuous Deployment
The New Stack
Kris Nova of Deis at CouldNativecon/Kubecon in Berlin
The New Stack
Docker's Quest for Simplicity with the Evolution of Containerd
The New Stack
Developers First: The Cloud Foundry Service Broker API and Kubernetes
The New Stack
Mapping the Future of CoreOS's rkt in the CNCF
The New Stack
Red Hat and Dell EMC: Two Perspectives from DockerCon
The New Stack
Capital One Opened its APIs to Third-Party Developers — Here’s What They Learned
The New Stack
SUSE Joins the CNCF, Brings Kubernetes to OpenStack Cloud 7
The New Stack
How Capital One Brings Open Source To The Banking Industry
The New Stack
OSCON Is Coming Back To Portland, A Show Wrapup With Co-Chair Kelsey Hightower
The New Stack
Dev Or Ops Doesn’t Matter, You Need Observability
The New Stack
Taking The Next Steps In Developing An Open Source Culture
The New Stack
SXSW 2017: How Capital One Became Technology-First With Open Source
The New Stack
Apcera Old Apps Spanning New Clouds
The New Stack
Provenance: The Peace of Mind Chef Habitat Seeks to Deliver
The New Stack
InSpec: Human Readable, Automated Compliance
The New Stack
The Evolution of SAP HANA Express
The New Stack
Women Engineers Who Inspire And Never Give Up
The New Stack
Three Perspectives on the Evolution of Container Security
The New Stack
More on: AI Systems Design
View skill →
Related AI Lessons
⚡
⚡
⚡
⚡
Meta Is in Crisis, Google Search’s Makeover, and AI Gets Booed by Graduates
Wired AI
What I Do Between Biotech Jobs, Part 1: The 20-Line Script That Outsmarted an AI
Medium · AI
Spotify and Universal Music strike deal allowing fan-made AI covers and remixes
TechCrunch AI
Google Just Turned Search Into Something It Has Never Been Before
Medium · AI
🎓
Tutor Explanation
DeepCamp AI