Azure Data engineer
- Ztek Consulting Inc
110 Days Ago
NA
NA
Des Moines-IA
10-15 Years
Required Skills: Azure ML and DatabricksAzure ML and Databricks, including hands-on experience in building and optimizing data pipelines.
Job Description
Design, develop, and maintain scalable data pipelines and ETL processes to support data science and analytics initiatives.
• Utilize Azure ML and Databricks to build and optimize data architectures and workflows.
• Collaborate with data scientists, analysts, and business stakeholders to understand data requirements and ensure data availability and quality.
• Implement data integration solutions to consolidate data from various sources into a unified data platform.
• Analyze the generation patterns from various Solar Generation facilities and help filed teams to identify the anomalies in generation.
• Ensure data security, privacy, and compliance with relevant regulations and best practices.
• Monitor and troubleshoot data pipelines and workflows to ensure reliability and performance.
• Stay up-to-date with the latest advancements in data engineering, big data technologies, and cloud platforms, and apply them to improve existing processes and systems.
Qualifications:
• Bachelor’s degree in computer science, Data Engineering, or a related field.
• 10+ years of experience in data engineering, data integration, and ETL processes.
• Extensive experience with Azure ML and Databricks, including hands-on experience in building and optimizing data pipelines.
• Proficiency in programming languages such as Python, SQL, and Scala.
• Strong understanding of data engineering principles and best practices.
• Experience with big data technologies such as Hadoop, Spark, and Kafka.
• Excellent problem-solving skills and the ability to work independently and as part of a team.
• Strong communication and interpersonal skills, with the ability to convey complex technical information to a non-technical audience.
• Experience in solar generation data analysis is a plus.
