Principal Data Engineer
  • VDart, Inc.
3 Days Ago
NA
W2, C2C
Houston-TX
15-17 Years
Required Skills: Azure Data Factory, ADF, Azure Data Lake Storage, ADLS, Azure Databricks, Databricks Delta Live Tables, DLT, Databricks Unity Catalog, Qlik Replicate, Azure Synapse Analytics, Azure Stream Analytics, SQL, Python, Scala, Spark SQL, Structured Streaming, ETL Pipelines, Data Lake Architecture, Data Governance, Cloud Data Migration
Job Description

We are seeking an experienced Principal Data Engineer to lead the design, development, and implementation of scalable cloud-based data engineering and analytics solutions. The ideal candidate will have strong expertise in Microsoft Azure, Databricks, data lake architecture, data governance, and large-scale data processing. This role requires a combination of hands-on technical expertise, architecture capabilities, and technical leadership to drive data engineering initiatives, optimize performance, and ensure data quality and governance across enterprise environments.

Responsibilities

  • Architect, design, and implement scalable Azure-based data lake solutions.
  • Lead the development of data transformation pipelines using Databricks Delta Live Tables and Structured Streaming.
  • Establish technical roadmaps for data engineering capabilities using Qlik Replicate, Databricks, and Unity Catalog.
  • Optimize Databricks resources and workloads for high-volume data processing and cost efficiency.
  • Implement data governance, lineage, metadata management, and data quality standards using Unity Catalog.
  • Develop and optimize scalable ETL pipelines and data workflows.
  • Lead data migration initiatives from on-premises environments to cloud platforms and between cloud environments.
  • Monitor data pipeline performance and troubleshoot production issues.
  • Ensure compliance, security, and data governance best practices.
  • Collaborate with business and technical stakeholders to align data engineering initiatives with organizational objectives.
  • Provide technical leadership, mentor engineering teams, and facilitate knowledge-sharing sessions.

Requirements

Required Technical Skills
  • Strong expertise in Microsoft Azure data services.
  • Hands-on experience with Azure Data Factory (ADF), Azure Data Lake Storage (ADLS), Azure Databricks, Azure Stream Analytics, and Azure Synapse Analytics.
  • Advanced proficiency in SQL and programming languages such as Python, Scala, and Spark SQL.
  • Strong experience with Databricks Delta Live Tables and Structured Streaming.
  • Expertise in Databricks Unity Catalog, data governance, data lineage, and metadata management.
  • Experience with Qlik Replicate.
  • Strong understanding of data lake architecture and enterprise data engineering.
  • Experience designing and optimizing large-scale ETL pipelines.
  • Expertise in cloud data migration and performance optimization.
Preferred Qualifications
  • Bachelor's or Master's degree in IT, Computer Science, or a related discipline.
  • 15–17 years of overall IT experience.
  • Proven experience delivering enterprise-scale cloud data analytics solutions.
  • Azure Data Engineering or Databricks certifications are preferred.
  • Experience with Hadoop, Kafka, and Apache Spark.
  • Familiarity with Power BI or Qlik Sense.
  • Knowledge of AI/ML, NLP, and Generative AI concepts.
  • Experience working with Agile and Kanban methodologies.
Leadership & Communication Skills
  • Strong technical leadership and team management capabilities.
  • Excellent communication and stakeholder management skills.
  • Ability to translate complex technical challenges into business solutions.
  • Strong ownership, accountability, and problem-solving abilities.
  • Experience identifying technical risks and implementing mitigation strategies.

Jobseeker

Looking For Job?
Search Jobs

Recruiter

Are You Recruiting?
Search Candidates