We are seeking an experienced Senior Azure Databricks & PySpark Data Engineer / Architect to design, develop, and implement enterprise-scale Azure Data Engineering solutions.

The ideal candidate must have strong hands-on experience with Azure Databricks, PySpark, Azure Data Factory, and Azure Cloud services, along with expertise in modern Lakehouse architecture and DevOps practices.

The candidate will be responsible for building scalable data engineering solutions, developing data pipelines, implementing cloud-native technologies, and driving engineering best practices within healthcare-focused enterprise environments.

Key Responsibilities

• Design, develop, and maintain enterprise-scale Azure Data Engineering solutions.
• Build and optimize data pipelines using Azure Databricks, PySpark, and Azure Data Factory.
• Develop and maintain modern Lakehouse architectures using Azure Synapse and Databricks.
• Implement and manage Databricks workflows and Unity Catalog.
• Build scalable data processing solutions using ADLS Gen2 and Azure SQL.
• Develop Python packages and support package builds.
• Implement DevOps practices and CI/CD processes for data engineering solutions.
• Develop and maintain data architecture and low-level design documents.
• Collaborate with internal engineering teams and customers to deliver high-quality solutions.
• Follow standard coding principles, guidelines, and engineering best practices.
• Lead technical decisions, mentor team members, and drive engineering improvements.

Mandatory Technical Skills

• 4+ years of hands-on experience with Azure Databricks and PySpark.
• 2+ years of experience with Databricks Workflows and Unity Catalog.
• 3+ years of experience with Azure Data Factory (ADF).
• 3+ years of experience with Azure Data Lake Storage Gen2 (ADLS Gen2).
• 3+ years of experience with Azure SQL.
• 2+ years of experience with Python programming and package builds.
• 5+ years of experience with the Azure Cloud platform.
• Strong understanding of Azure Synapse and Databricks Lakehouse architecture.
• Strong communication and collaboration skills.

Preferred / Nice-to-Have Skills

• Change Data Capture (CDC) solutions, preferably Debezium.
• Azure Entra ID / Active Directory.
• GitHub Actions.
• Workflow orchestration tools such as Airflow, Dagster, and Azure Logic Apps.
• Event-driven architecture using Kafka and Azure Event Hubs.
• Google Cloud Pub/Sub.
• Google Cloud Storage (GCS).
• Experience managing cloud storage solutions on Azure Data Lake Storage.

Required Qualifications

• 10–20 years of overall IT experience.
• Strong hands-on Azure Data Engineering background.
• Proven experience developing enterprise-scale data pipelines and Lakehouse solutions.
• Ability to prepare low-level design documents and follow coding standards.
• Strong technical communication and collaboration skills.
• Ability to take ownership of solution design and technical decisions.