PySpark on Kubernetes -- Avoiding the pain of YARN

About this talk

Apache Spark is one of the most popular big data tools, and starting last year has had integrated support for running on Kubernetes. This talk will introduce some of the use cases of Apache Spark quickly (machine learning, ETL, etc.) and then look at the current cluster managers Spark runs on and their limitations. Most of the focus will be around running Python code, and the challenges associated with dependencies along with general challenges like scale-up & down. It’s not all sunshine and roses though, I will talk about some of the limitations of our current approach and the work being done to improve this.