Data Engineering with Python and AI/LLMs – Data Loading Tutorial
Master data ingestion for data engineering with Python. Learn to tackle common pipeline failures like schema changes and API limits by adopting the mindset and practices of a senior platform engineer. This course covers essential techniques including extracting data from APIs, automatic schema management, incremental loading, and orchestrating scalable, automated workflows using modern tools.
Course developed by Alexey Grigorev & Adrian Brudaru.
💻 Code: https://github.com/dlt-hub/dlthub-education/tree/main/courses/freecodecamp/de_with_dlt_2025
🏗️ dlthub.com provided a grant to make this course possible.
⭐️ Contents ⭐️
Alexey's part
0:00:00 1. Introduction
0:08:02 2. What is data ingestion
0:10:04 3. Extracting data: Data Streaming & Batching
0:14:00 4. Extracting data: Working with RestAPI
0:29:36 5. Normalizing data
0:43:41 6. Loading data into DuckDB
0:48:39 7. Dynamic schema management
0:56:26 8. What is next?
Adrian's part
0:56:36 1. Introduction
0:59:29 2. Overview
1:02:08 3. Extracting data with dlt: dlt RestAPI Client
1:08:05 4. dlt Resources
1:10:42 5. How to configure secrets
1:15:12 6. Normalizing data with dlt
1:24:09 7. Data Contracts
1:31:05 8. Alerting schema changes
1:33:56 9. Loading data with dlt
1:33:56 10. Write dispositions
1:37:34 11. Incremental loading
1:43:46 12. Loading data from SQL database to SQL database
1:47:46 13. Backfilling
1:50:42 14. SCD2
1:54:29 15. Performance tuning
2:03:12 16. Loading data to Data Lakes & Lakehouses & Catalogs
2:12:17 17. Loading data to Warehouses/MPPs,Staging
2:18:15 18. Deployment & orchestration
2:18:15 19. Deployment with Git Actions
2:29:04 20. Deployment with Crontab
2:40:05 21. Deployment with Dagster
2:49:47 22. Deployment with Airflow
3:07:00 23. Create pipelines with LLMs: Understanding the challenge
3:10:35 24. Create pipelines with LLMs: Creating prompts and LLM friendly documentation
3:31:38 25. Create pipelines with LLMs: Demo
🎉 Thanks to our Champion and Sponsor supporters:
👾 Drake Mil
Watch on YouTube ↗
(saves to browser)
Sign in to unlock AI tutor explanation · ⚡30
Playlist
Uploads from freeCodeCamp.org · freeCodeCamp.org · 0 of 60
← Previous
Next →
1
2
3
4
5
6
7
8
9
10
11
12
13
14
15
16
17
18
19
20
21
22
23
24
25
26
27
28
29
30
31
32
33
34
35
36
37
38
39
40
41
42
43
44
45
46
47
48
49
50
51
52
53
54
55
56
57
58
59
60
React: Production Server Setup Part 2 - Live Coding with Jesse
freeCodeCamp.org
cookies vs localStorage vs sessionStorage - Beau teaches JavaScript
freeCodeCamp.org
Browser history tutorial - Beau teaches JavaScript
freeCodeCamp.org
Graph Data Structure Intro (inc. adjacency list, adjacency matrix, incidence matrix)
freeCodeCamp.org
React: Parameterized Routing with Next.js - Live Coding with Jesse
freeCodeCamp.org
React: Dealing with jQuery Issues - Live Coding with Jesse
freeCodeCamp.org
setInterval and setTimeout: timing events - Beau teaches JavaScript
freeCodeCamp.org
Browser and Device Testing - Live Coding with Jesse
freeCodeCamp.org
Last Minute Updates - Live Coding with Jesse
freeCodeCamp.org
Post Launch Updates - Live Coding with Jesse
freeCodeCamp.org
React: Setting Up Google Analytics - Live Coding with Jesse
freeCodeCamp.org
React: Masonry Layout - Live Coding with Jesse
freeCodeCamp.org
Load Balancing Digital Ocean Droplets - Live Coding with Jesse
freeCodeCamp.org
try, catch, finally, throw - error handling in JavaScript
freeCodeCamp.org
Load Balancing: SSL Passthrough Setup - Live Coding with Jesse
freeCodeCamp.org
Graphs: breadth-first search - Beau teaches JavaScript
freeCodeCamp.org
React: Masonry Layout Part 2 - Live Coding with Jesse
freeCodeCamp.org
React: WordPress API Live Search - Live Coding with Jesse
freeCodeCamp.org
Creating WordPress Custom Post Types - Live Coding With Jesse
freeCodeCamp.org
Dates - Beau teaches JavaScript
freeCodeCamp.org
Miscellaneous Front End Updates - Live Coding with Jesse
freeCodeCamp.org
Merging a Pull Request from GitHub - Live Coding with Jesse
freeCodeCamp.org
React + Prettier + Standard JS - Live Coding with Jesse
freeCodeCamp.org
React: Sortable Responsive Table - Live Coding with Jesse
freeCodeCamp.org
Geolocation Sorting by Distance - Live Coding with Jesse
freeCodeCamp.org
Tradeoff Matrix - Agile Software Development
freeCodeCamp.org
The Definition of Ready - Agile Software Development
freeCodeCamp.org
Getting first React job without experience - Ask Preethi
freeCodeCamp.org
React: Google Analytics Click Tracking - Live Coding with Jesse
freeCodeCamp.org
Submitting a PR to an Open Source Project - Live Coding with Jesse
freeCodeCamp.org
Should I go back to school to get CS degree? - Ask Preethi
freeCodeCamp.org
Hero Section CSS Changes - Live Coding with Jesse
freeCodeCamp.org
Working Agreement - Agile Software Development
freeCodeCamp.org
A day at Pennybox with Co-Founder Reji Eapen
freeCodeCamp.org
React: Sorting and Filtering Data - Live Coding with Jesse
freeCodeCamp.org
React: Sorting and Filtering Data Part 2 - Live Coding with Jesse
freeCodeCamp.org
React: Building a New UI - Live Coding with Jesse
freeCodeCamp.org
Definition of Done - Agile Software Development
freeCodeCamp.org
Getting started with jQuery (tutorial) - Beau teaches JavaScript
freeCodeCamp.org
Making a React Blog with WordPress Content - Live Coding with Jesse
freeCodeCamp.org
React, NextJS, CSS - Live Coding with Jesse
freeCodeCamp.org
jQuery events - Beau teaches JavaScript
freeCodeCamp.org
React/NextJS Routing and WordPress API Custom Types - Live Coding with Jesse
freeCodeCamp.org
React: Working with API Data - Live Coding with Jesse
freeCodeCamp.org
React: Refactoring Components - Live Streaming with Jesse
freeCodeCamp.org
jQuery effects - Beau teaches JavaScript
freeCodeCamp.org
More React Refactoring - Live Coding with Jesse
freeCodeCamp.org
animate in jQuery - Beau teaches JavaScript
freeCodeCamp.org
"Finishing" My React Site - Live Coding with Jesse
freeCodeCamp.org
Starting a New React Project (P2D1) - Live Coding with Jesse
freeCodeCamp.org
React Project 2 Day 2: Learning Material UI - Live Coding with Jesse
freeCodeCamp.org
The Agile Manifesto - Agile Software Development
freeCodeCamp.org
jQuery: get and set with http, text, val, and attr - Beau teaches JavaScript
freeCodeCamp.org
React Project 2 Day 3 - Live Coding with Jesse
freeCodeCamp.org
The INVEST approach to product backlog items
freeCodeCamp.org
React Project 2 Day 4 - Live Coding with Jesse
freeCodeCamp.org
Chickens and Pigs - Agile Software Development
freeCodeCamp.org
React Project 2 Day 5 - Live Coding with Jesse
freeCodeCamp.org
jQuery: add and remove DOM elements - Beau teaches JavaScript
freeCodeCamp.org
React Project 2 Day 6 - Live Coding with Jesse
freeCodeCamp.org
More on: Data Literacy
View skill →
Related AI Lessons
⚡
⚡
⚡
⚡
I Found a Tool That Might Save Fellow Gen-Xers From the Blank Screen Problem
Medium · AI
Apple operating systems focus on optimization and AI for 2027
Medium · AI
4 Simple AI Workflows That Gen Z Is Using to Build Internet Riches
Medium · AI
In 2 years most people won’t need separate AI tools, it’ll all just be built into your OS. Agree or disagree?
Reddit r/artificial
Chapters (33)
1. Introduction
8:02
2. What is data ingestion
10:04
3. Extracting data: Data Streaming & Batching
14:00
4. Extracting data: Working with RestAPI
29:36
5. Normalizing data
43:41
6. Loading data into DuckDB
48:39
7. Dynamic schema management
56:26
8. What is next?
56:36
1. Introduction
59:29
2. Overview
1:02:08
3. Extracting data with dlt: dlt RestAPI Client
1:08:05
4. dlt Resources
1:10:42
5. How to configure secrets
1:15:12
6. Normalizing data with dlt
1:24:09
7. Data Contracts
1:31:05
8. Alerting schema changes
1:33:56
9. Loading data with dlt
1:33:56
10. Write dispositions
1:37:34
11. Incremental loading
1:43:46
12. Loading data from SQL database to SQL database
1:47:46
13. Backfilling
1:50:42
14. SCD2
1:54:29
15. Performance tuning
2:03:12
16. Loading data to Data Lakes & Lakehouses & Catalogs
2:12:17
17. Loading data to Warehouses/MPPs,Staging
2:18:15
18. Deployment & orchestration
2:18:15
19. Deployment with Git Actions
2:29:04
20. Deployment with Crontab
2:40:05
21. Deployment with Dagster
2:49:47
22. Deployment with Airflow
3:07:00
23. Create pipelines with LLMs: Understanding the challenge
3:10:35
24. Create pipelines with LLMs: Creating prompts and LLM friendly documentation
3:31:38
25. Create pipelines with LLMs: Demo
🎓
Tutor Explanation
DeepCamp AI