Data Engineering Course for Beginners

freeCodeCamp.org · Beginner ·🔄 Data Engineering ·2y ago

Key Takeaways

This video course covers the essentials of data engineering, including databases, Docker, analytical engineering, data pipeline building with Airflow, batch processing with Spark, and streaming data with Kafka.

Original Description

Learn the essentials of data engineering in this course for beginners. You'll learn about Databases, Docker, and analytical engineering. You'll explore advanced topics like data pipeline building with Airflow, and engage in batch processing with Spark and streaming data with Kafka. The course culminates in a comprehensive project, putting your skills to the test in creating a full end-to-end pipeline. ✏️ Justin Chau created this course. Course Resources: https://transparent-trout-f2f.notion.site/FreeCodeCamp-Data-Engineering-Course-Resources-e9d2b97aed5b4d4a922257d953c4e759 Thanks to Airbyte for providing a grant to make this course possible. ❤️ Try interactive Databases courses we love, right in your browser: https://scrimba.com/freeCodeCamp-Databases (Made possible by a grant from our friends at Scrimba) ⭐️ Contents ⭐️ ⌨️ (0:00:00) Introduction ⌨️ (0:00:36) Why Data Engineering ⌨️ (0:03:14) Docker ⌨️ (0:30:38) SQL ⌨️ (1:04:32) Building a Data Pipeline from Scratch ⌨️ (1:31:03) dbt ⌨️ (2:04:11) CRON Job ⌨️ (2:07:54) Airflow ⌨️ (2:41:14) Airbyte ⌨️ (3:01:54) Outro 🎉 Thanks to our Champion and Sponsor supporters: 👾 davthecoder 👾 jedi-or-sith 👾 南宮千影 👾 Agustín Kussrow 👾 Nattira Maneerat 👾 Heather Wcislo 👾 Serhiy Kalinets 👾 Justin Hual 👾 Otis Morgan 👾 Oscar Rahnama -- Learn to code for free and get a developer job: https://www.freecodecamp.org Read hundreds of articles on programming: https://freecodecamp.org/news
Watch on YouTube ↗ (saves to browser)
Sign in to unlock AI tutor explanation · ⚡30

Playlist

Uploads from freeCodeCamp.org · freeCodeCamp.org · 0 of 60

← Previous Next →
1 React: Production Server Setup Part 2 - Live Coding with Jesse
React: Production Server Setup Part 2 - Live Coding with Jesse
freeCodeCamp.org
2 cookies vs localStorage vs sessionStorage - Beau teaches JavaScript
cookies vs localStorage vs sessionStorage - Beau teaches JavaScript
freeCodeCamp.org
3 Browser history tutorial - Beau teaches JavaScript
Browser history tutorial - Beau teaches JavaScript
freeCodeCamp.org
4 Graph Data Structure Intro (inc. adjacency list, adjacency matrix, incidence matrix)
Graph Data Structure Intro (inc. adjacency list, adjacency matrix, incidence matrix)
freeCodeCamp.org
5 React: Parameterized Routing with Next.js - Live Coding with Jesse
React: Parameterized Routing with Next.js - Live Coding with Jesse
freeCodeCamp.org
6 React: Dealing with jQuery Issues - Live Coding with Jesse
React: Dealing with jQuery Issues - Live Coding with Jesse
freeCodeCamp.org
7 setInterval and setTimeout: timing events - Beau teaches JavaScript
setInterval and setTimeout: timing events - Beau teaches JavaScript
freeCodeCamp.org
8 Browser and Device Testing - Live Coding with Jesse
Browser and Device Testing - Live Coding with Jesse
freeCodeCamp.org
9 Last Minute Updates - Live Coding with Jesse
Last Minute Updates - Live Coding with Jesse
freeCodeCamp.org
10 Post Launch Updates - Live Coding with Jesse
Post Launch Updates - Live Coding with Jesse
freeCodeCamp.org
11 React: Setting Up Google Analytics - Live Coding with Jesse
React: Setting Up Google Analytics - Live Coding with Jesse
freeCodeCamp.org
12 React: Masonry Layout - Live Coding with Jesse
React: Masonry Layout - Live Coding with Jesse
freeCodeCamp.org
13 Load Balancing Digital Ocean Droplets - Live Coding with Jesse
Load Balancing Digital Ocean Droplets - Live Coding with Jesse
freeCodeCamp.org
14 try, catch, finally, throw - error handling in JavaScript
try, catch, finally, throw - error handling in JavaScript
freeCodeCamp.org
15 Load Balancing: SSL Passthrough Setup - Live Coding with Jesse
Load Balancing: SSL Passthrough Setup - Live Coding with Jesse
freeCodeCamp.org
16 Graphs: breadth-first search - Beau teaches JavaScript
Graphs: breadth-first search - Beau teaches JavaScript
freeCodeCamp.org
17 React: Masonry Layout Part 2 - Live Coding with Jesse
React: Masonry Layout Part 2 - Live Coding with Jesse
freeCodeCamp.org
18 React: WordPress API Live Search - Live Coding with Jesse
React: WordPress API Live Search - Live Coding with Jesse
freeCodeCamp.org
19 Creating WordPress Custom Post Types - Live Coding With Jesse
Creating WordPress Custom Post Types - Live Coding With Jesse
freeCodeCamp.org
20 Dates - Beau teaches JavaScript
Dates - Beau teaches JavaScript
freeCodeCamp.org
21 Miscellaneous Front End Updates - Live Coding with Jesse
Miscellaneous Front End Updates - Live Coding with Jesse
freeCodeCamp.org
22 Merging a Pull Request from GitHub - Live Coding with Jesse
Merging a Pull Request from GitHub - Live Coding with Jesse
freeCodeCamp.org
23 React + Prettier + Standard JS - Live Coding with Jesse
React + Prettier + Standard JS - Live Coding with Jesse
freeCodeCamp.org
24 React: Sortable Responsive Table - Live Coding with Jesse
React: Sortable Responsive Table - Live Coding with Jesse
freeCodeCamp.org
25 Geolocation Sorting by Distance - Live Coding with Jesse
Geolocation Sorting by Distance - Live Coding with Jesse
freeCodeCamp.org
26 Tradeoff Matrix - Agile Software Development
Tradeoff Matrix - Agile Software Development
freeCodeCamp.org
27 The Definition of Ready - Agile Software Development
The Definition of Ready - Agile Software Development
freeCodeCamp.org
28 Getting first React job without experience - Ask Preethi
Getting first React job without experience - Ask Preethi
freeCodeCamp.org
29 React: Google Analytics Click Tracking - Live Coding with Jesse
React: Google Analytics Click Tracking - Live Coding with Jesse
freeCodeCamp.org
30 Submitting a PR to an Open Source Project - Live Coding with Jesse
Submitting a PR to an Open Source Project - Live Coding with Jesse
freeCodeCamp.org
31 Should I go back to school to get CS degree? - Ask Preethi
Should I go back to school to get CS degree? - Ask Preethi
freeCodeCamp.org
32 Hero Section CSS Changes - Live Coding with Jesse
Hero Section CSS Changes - Live Coding with Jesse
freeCodeCamp.org
33 Working Agreement - Agile Software Development
Working Agreement - Agile Software Development
freeCodeCamp.org
34 A day at Pennybox with Co-Founder Reji Eapen
A day at Pennybox with Co-Founder Reji Eapen
freeCodeCamp.org
35 React: Sorting and Filtering Data - Live Coding with Jesse
React: Sorting and Filtering Data - Live Coding with Jesse
freeCodeCamp.org
36 React: Sorting and Filtering Data Part 2 - Live Coding with Jesse
React: Sorting and Filtering Data Part 2 - Live Coding with Jesse
freeCodeCamp.org
37 React: Building a New UI - Live Coding with Jesse
React: Building a New UI - Live Coding with Jesse
freeCodeCamp.org
38 Definition of Done - Agile Software Development
Definition of Done - Agile Software Development
freeCodeCamp.org
39 Getting started with jQuery (tutorial) - Beau teaches JavaScript
Getting started with jQuery (tutorial) - Beau teaches JavaScript
freeCodeCamp.org
40 Making a React Blog with WordPress Content - Live Coding with Jesse
Making a React Blog with WordPress Content - Live Coding with Jesse
freeCodeCamp.org
41 React, NextJS, CSS - Live Coding with Jesse
React, NextJS, CSS - Live Coding with Jesse
freeCodeCamp.org
42 jQuery events - Beau teaches JavaScript
jQuery events - Beau teaches JavaScript
freeCodeCamp.org
43 React/NextJS Routing and WordPress API Custom Types - Live Coding with Jesse
React/NextJS Routing and WordPress API Custom Types - Live Coding with Jesse
freeCodeCamp.org
44 React: Working with API Data - Live Coding with Jesse
React: Working with API Data - Live Coding with Jesse
freeCodeCamp.org
45 React: Refactoring Components - Live Streaming with Jesse
React: Refactoring Components - Live Streaming with Jesse
freeCodeCamp.org
46 jQuery effects - Beau teaches JavaScript
jQuery effects - Beau teaches JavaScript
freeCodeCamp.org
47 More React Refactoring - Live Coding with Jesse
More React Refactoring - Live Coding with Jesse
freeCodeCamp.org
48 animate in jQuery - Beau teaches JavaScript
animate in jQuery - Beau teaches JavaScript
freeCodeCamp.org
49 "Finishing" My React Site - Live Coding with Jesse
"Finishing" My React Site - Live Coding with Jesse
freeCodeCamp.org
50 Starting a New React Project (P2D1) - Live Coding with Jesse
Starting a New React Project (P2D1) - Live Coding with Jesse
freeCodeCamp.org
51 React Project 2 Day 2: Learning Material UI - Live Coding with Jesse
React Project 2 Day 2: Learning Material UI - Live Coding with Jesse
freeCodeCamp.org
52 The Agile Manifesto - Agile Software Development
The Agile Manifesto - Agile Software Development
freeCodeCamp.org
53 jQuery: get and set with http, text, val, and attr - Beau teaches JavaScript
jQuery: get and set with http, text, val, and attr - Beau teaches JavaScript
freeCodeCamp.org
54 React Project 2 Day 3 - Live Coding with Jesse
React Project 2 Day 3 - Live Coding with Jesse
freeCodeCamp.org
55 The INVEST approach to product backlog items
The INVEST approach to product backlog items
freeCodeCamp.org
56 React Project 2 Day 4 - Live Coding with Jesse
React Project 2 Day 4 - Live Coding with Jesse
freeCodeCamp.org
57 Chickens and Pigs - Agile Software Development
Chickens and Pigs - Agile Software Development
freeCodeCamp.org
58 React Project 2 Day 5 - Live Coding with Jesse
React Project 2 Day 5 - Live Coding with Jesse
freeCodeCamp.org
59 jQuery: add and remove DOM elements - Beau teaches JavaScript
jQuery: add and remove DOM elements - Beau teaches JavaScript
freeCodeCamp.org
60 React Project 2 Day 6 - Live Coding with Jesse
React Project 2 Day 6 - Live Coding with Jesse
freeCodeCamp.org

This course teaches the fundamentals of data engineering, covering topics such as databases, Docker, and data pipeline building. Students will learn to build a data pipeline from scratch and engage in batch processing and streaming data. The course culminates in a comprehensive project, putting skills to the test.

Key Takeaways
  1. Install Docker
  2. Create a database
  3. Build a data pipeline with Airflow
  4. Process batch data with Spark
  5. Stream data with Kafka
  6. Write SQL queries
  7. Use dbt for data transformation
💡 Data engineering is a crucial step in the data analytics process, and building a data pipeline from scratch requires a thorough understanding of databases, data processing, and streaming data.

Related Reads

📰
What Can We Do When Memory Becomes the New Bottleneck in Data Engineering?
Learn how to overcome memory bottlenecks in data engineering using Pandas chunking, Dask, and Polars, and why it matters for processing large datasets
Towards Data Science
📰
Migrate from Ponder to Envio HyperIndex
Learn to migrate your indexer from Ponder to Envio HyperIndex to scale your data management
Dev.to · Envio
📰
Data Backfilling with Apache Airflow: Architectures and Implementations for Historical Data Processing
Learn how to implement data backfilling with Apache Airflow for historical data processing and improve your data pipeline's accuracy and reliability
Dev.to · Wangila russell
📰
Building a Production-Style Weather Analytics Pipeline from Scratch: ETL, ELT, Star Schema, and…
Learn to build a production-ready weather analytics pipeline from scratch using Python, DuckDB, and Apache tools, and understand the importance of ETL, ELT, and Star Schema in data engineering
Medium · Python
Up next
A Moment Frozen in Time | Arnav Iyengar | TEDxJenks Youth
TEDx Talks
Watch →