Responsibilities

• Design and develop scalable data pipelines using Azure Databricks, Apache Spark, PySpark, Python, and Spark SQL.
• Build and maintain Lakehouse architectures using Delta Lake, Delta Tables, Unity Catalog, and Medallion Architecture (Bronze/Silver/Gold).
• Develop reusable batch and streaming ETL/ELT pipelines using Databricks Workflows, Lakeflow/Delta Live Tables, and Structured Streaming.
• Optimize Spark workloads through partitioning, caching, query optimization, join/shuffle optimization, and performance tuning.
• Develop complex SQL transformations, data models, schemas, and curated datasets for analytics and downstream applications.
• Implement data quality, security, access controls, lineage, monitoring, and error-handling capabilities.
• Build production-ready solutions following engineering best practices including Git, CI/CD, automated testing, code reviews, and deployment automation.
• Collaborate with architects, analysts, data scientists, and business teams to deliver reliable enterprise data solutions.

Required Skills

• Bachelor’s degree in Computer Science, Information Technology, Engineering, Data Science, or a related technical field. 7+ years of overall data engineering/software engineering experience, with significant hands-on experience in large-scale data platforms.
• 3+ years of hands-on Azure Databricks experience, including production implementations.
• Strong hands-on expertise in Python, PySpark, Spark SQL, Apache Spark, and SQL.
• Proven experience with Delta Lake, Unity Catalog, Medallion Architecture, Databricks Workflows/Lakeflow, and Structured Streaming.
• Experience designing and supporting production-grade ETL/ELT pipelines processing large volumes of data.
• Strong understanding of data modeling, performance optimization, data quality, security, governance, and monitoring.
• Experience with Git, CI/CD, automated testing, and Agile engineering practices.
• Databricks certifications such as Databricks Certified Data Engineer Associate/Professional are preferred but not required.

Preferred

• Experience with ADLS Gen2, Azure Data Factory, Azure SQL, or other Azure data services
• Databricks certification
• Experience with infrastructure/deployment automation and enterprise data governance