Databricks Data Engineer
- Aneka Talent Solutions
13 Hours Ago
60-70 per Hourly
W2
Charlotte NC-NC
3-7 Years
Required Skills: Databricks, Apache Spark, PySpark, Python, SQL, Azure Data Factory, Azure Data Lake Storage Gen2
Job Description
We are looking for an experienced Azure Databricks Engineer to design, build, and maintain scalable data pipelines, lakehouse architectures, and analytics solutions using Microsoft Azure and the Databricks platform. The ideal candidate should have strong expertise in big data processing, cloud-based data engineering, and data pipeline optimization.
Key Responsibilities
- Pipeline Development: Design and build ETL and ELT data pipelines using Azure Data Factory (ADF) and Azure Databricks.
- Big Data Processing: Develop optimized data processing solutions using Apache Spark, PySpark, Python, Scala, and advanced SQL.
- Storage & Architecture: Implement and manage data lakes using Azure Data Lake Storage Gen2 (ADLS Gen2), Delta Lake, and lakehouse architecture patterns.
- Governance & Security: Configure data governance, data lineage, and access controls using Databricks Unity Catalog and Azure IAM.
- Optimization & Operations: Monitor, troubleshoot, and optimize Spark jobs and data pipelines for performance, reliability, and cost efficiency.
- CI/CD & DevOps: Support deployment processes using Azure DevOps, Git version control, and Infrastructure as Code (IaC) tools such as Terraform.
Required Skills & Qualifications
- Experience: 3 to 7+ years of experience in data engineering or cloud-based data platform development.
- Core Technologies: Hands-on experience with Azure Databricks, Apache Spark, PySpark, and Azure Data Factory (ADF).
- Programming Languages: Advanced SQL skills and solid programming knowledge in Python or Scala.
- Data Engineering Concepts: Strong understanding of Delta Lake, relational and NoSQL data storage, data modeling, and data quality standards.
- Cloud & Storage: Experience with Microsoft Azure and Azure Data Lake Storage Gen2 (ADLS Gen2).
- DevOps Tools: Familiarity with Azure DevOps, Git, and Terraform.
Education
- Bachelor's degree in Computer Science, Engineering, or a related quantitative field is preferred.
