• Experience with Apache Spark framework
• Hands-on experience in building ETL/ELT solutions using Spark SQL
• Experience working with Spark RDD’s and DataFrames
• Experience working with Azure Databricks is a must
• Design, develop, test, deploy, maintain and improve data integration pipeline
• Strong analytical experience with databases in writing complex queries, query optimization, debugging, user defined functions, views, indexes etc.
• Experience in data wrangling and strong SQL skills
• Experience working with HDFS file system, parquet, and Delta file formats
• Experience in Spark optimizing queries
• Experience working with ADF is a major plus