Data Engineering with Python and AI/LLMs – Data Loading Tutorial

Post Content

Master data ingestion for data engineering with Python. Learn to tackle common pipeline failures like schema changes and API limits by adopting the mindset and practices of a senior platform engineer. This course covers essential techniques including extracting data from APIs, automatic schema management, incremental loading, and orchestrating scalable, automated workflows using modern tools.

Course developed by Alexey Grigorev & Adrian Brudaru.

Code: https://github.com/dlt-hub/dlthub-education/tree/main/courses/freecodecamp/de_with_dlt_2025

dlthub.com provided a grant to make this course possible.

Contents
Alexey’s part
0:00:00 1. Introduction
0:08:02 2. What is data ingestion
0:10:04 3. Extracting data: Data Streaming & Batching
0:14:00 4. Extracting data: Working with RestAPI
0:29:36 5. Normalizing data
0:43:41 6. Loading data into DuckDB
0:48:39 7. Dynamic schema management
0:56:26 8. What is next?

Adrian’s part
0:56:36 1. Introduction
0:59:29 2. Overview
1:02:08 3. Extracting data with dlt: dlt RestAPI Client
1:08:05 4. dlt Resources
1:10:42 5. How to configure secrets
1:15:12 6. Normalizing data with dlt
1:24:09 7. Data Contracts
1:31:05 8. Alerting schema changes
1:33:56 9. Loading data with dlt
1:33:56 10. Write dispositions
1:37:34 11. Incremental loading
1:43:46 12. Loading data from SQL database to SQL database
1:47:46 13. Backfilling
1:50:42 14. SCD2
1:54:29 15. Performance tuning
2:03:12 16. Loading data to Data Lakes & Lakehouses & Catalogs
2:12:17 17. Loading data to Warehouses/MPPs,Staging
2:18:15 18. Deployment & orchestration
2:18:15 19. Deployment with Git Actions
2:29:04 20. Deployment with Crontab
2:40:05 21. Deployment with Dagster
2:49:47 22. Deployment with Airflow
3:07:00 23. Create pipelines with LLMs: Understanding the challenge
3:10:35 24. Create pipelines with LLMs: Creating prompts and LLM friendly documentation
3:31:38 25. Create pipelines with LLMs: Demo

Thanks to our Champion and Sponsor supporters:
Drake Milly
Ulises Moralez
Goddard Tan
David MG
Matthew Springman
Claudio
Oscar R.
jedi-or-sith
Nattira Maneerat
Justin Hual