Data Engineer Lead
- Hirextra Inc.
2 Days Ago
NA
W2
Atlanta-GA
8-15 Years
Required Skills: Databricks, Apache Spark, PySpark, SQL, Control-M
Job Description
Data Engineer Lead – Databricks & Control-M
We are seeking an experienced Data Engineer Lead with strong hands-on expertise in Databricks, Apache Spark, PySpark, SQL, and Control-M. The ideal candidate will lead data engineering initiatives, guide engineering teams, develop scalable data pipelines, and manage enterprise-level workflow orchestration.
Strong communication, leadership, stakeholder-management, and problem-solving skills are essential. Experience in the banking/financial services industry, particularly with CCAR and CECL reporting, is highly preferred.
Key Responsibilities
- Lead and mentor a team of data engineers and technical resources.
- Assign tasks, review deliverables, and ensure timely project execution.
- Design, develop, optimize, and maintain scalable data pipelines using Databricks and Apache Spark.
- Develop data processing solutions using PySpark, SQL, and Delta Lake.
- Build and manage enterprise workflows using Control-M.
- Configure job dependencies, scheduling, monitoring, alerts, and error-handling processes in Control-M.
- Troubleshoot data pipeline and workflow failures and drive root-cause resolution.
- Work closely with business, technology, and executive stakeholders.
- Translate complex technical concepts into clear business requirements and recommendations.
- Establish and enforce data-quality, governance, documentation, and development standards.
- Support cloud-based data architecture and ensure scalable, reliable solutions.
- Collaborate with architecture, application, infrastructure, risk, and compliance teams.
- Support regulatory and financial reporting data pipelines when required.
- Contribute to critical risk and compliance reporting frameworks, including CCAR and CECL, where applicable.
- Drive continuous improvement in data engineering processes, performance, reliability, and automation.
Required Qualifications
Leadership & Communication
- Minimum 2–3 years of experience leading data engineering teams or major technical projects.
- Strong team leadership and mentoring capabilities.
- Excellent verbal and written communication skills.
- Proven experience working directly with business and senior stakeholders.
- Strong analytical, troubleshooting, and decision-making skills.
Databricks & Data Engineering
- Strong hands-on experience with Databricks.
- Strong knowledge of Delta Lake.
- Advanced experience with PySpark / Apache Spark.
- Strong SQL development and optimization skills.
- Experience designing and maintaining scalable data pipelines.
- Knowledge of data engineering best practices, ETL/ELT, and data quality.
Control-M
- Deep hands-on expertise with Control-M.
- Experience creating and managing enterprise job schedules.
- Strong understanding of job dependencies and workflow orchestration.
- Experience with monitoring, alerts, failure recovery, and error handling.
- Ability to troubleshoot complex production workflows.
Cloud
Experience working with one or more major cloud platforms:
- AWS
- Microsoft Azure
- Google Cloud Platform (GCP)
Preferred Qualifications
- Banking or financial-services industry experience.
- Experience with CCAR (Comprehensive Capital Analysis and Review) reporting.
- Experience with CECL (Current Expected Credit Loss) reporting.
- Experience supporting risk, regulatory, compliance, or financial reporting data.
- Experience with enterprise-scale cloud data platforms.
- Knowledge of data governance, metadata, lineage, and data-quality frameworks.
Ideal Candidate
The ideal candidate is a hands-on Data Engineering Lead who can operate at both the technical and leadership levels. They should be comfortable developing Databricks/Spark pipelines, managing Control-M workflows, solving production issues, leading engineers, and communicating effectively with business and executive stakeholders.
Core Skills:
Databricks
Databricks
Delta Lake
PySpark
Apache Spark
SQL
Control-M
ETL/ELT
Cloud
Data Engineering
Technical Leadership
Stakeholder Management
Data Governance
CCAR
CECL
