Building a DAG Factory
- Speaker
- Albert Franzi Cros
- Event
- PyDay BCN (2021)
- Date
- Type
- Talk
- Language
- English
- Length
- 30 minutes
About this talk
At Typeform there are several teams that interact with data. Machine learning engineers, data scientists, data analysts, insights teams. None of them should have to worry about Airflow operators, upstream tasks, downstream tasks and all the million ways that writing Airflow DAGs by hand makes you miserable.
This is where the Data Platform teams comes to save the day and create Chester, a DAG factory, powered by a set of API connectors handled by us. Other teams only need to worry about where the data comes from and where it goes to, and write some YAML that describes it. All the rest is handled by Chester’s superpowers.
Photos (1)
