From Scripts to Systems: Scaling Your Python Analysis with Apache Airflow
- Speaker
- Marc Gonzalez Planas
- Event
- Python Meetup May
- Date
- Type
- Talk
- Language
- English
- Length
- 30 minutes
About this talk
Data Analysts often face a “production wall” where local Python scripts struggle to meet the demands of a live environment. Moving from ad-hoc analysis to scalable data products requires a shift in focus toward governance, observability, and idempotency. Without proper orchestration, we can fail.
Let’s learn how to bridge that gap by refactoring standard Python scripts into automated pipelines using Apache Airflow. We will explore the transition to an “orchestration” mindset, focusing on how to use Directed Acyclic Graphs (DAGs) and the TaskFlow API to transform modular Python functions into a an automatic system.