Required Skills: Databricks, PySpark, Python, Azure, GitHub Copilot, Claude
Job Description
Key Responsibilities
QA Strategy & Test Framework
- Own and evolve the end-to-end QA strategy for data pipelines, ETL/ELT workflows, and financial data integrations
- Design and implement scalable test frameworks covering data validation, schema integrity, transformation accuracy, and business rule compliance
- Define QA standards, best practices, and documentation requirements for the data engineering team
- Lead test planning, test case design, and execution across new pipeline builds and platform changes
Financial Data Validation & Reconciliation QA
- Validate the accuracy and completeness of wealth management datasets including positions, transactions, accounts, clients, advisors, and security master data
- Design and run reconciliation QA processes to surface breaks between custodians, internal systems, and third-party data providers
- Build automated data quality checks, threshold alerts, and validation rules to catch issues before they reach advisors or clients
- Investigate and document root causes of data quality failures and partner with engineering to drive permanent fixes
Pipeline & Integration Testing
- Lead QA efforts across data ingestion, transformation, and delivery layers within the Microsoft Azure and Databricks environment
- Design regression test suites to ensure pipeline changes don't introduce data quality regressions
- Collaborate with data engineers during development to shift quality left — embedding QA checkpoints earlier in the build cycle
- Validate data outputs against business requirements and financial data specifications
AI-Augmented QA
- Actively leverage AI tools (e.g., GitHub Copilot, Claude, ChatGPT) to accelerate test case generation, anomaly detection, and QA documentation
- Identify opportunities to apply AI/ML techniques to data quality problems such as automated break detection, outlier identification, or pattern-based validation
- Champion an AI-forward approach to QA across the team and bring practical recommendations for tooling improvements
Cross-Functional Collaboration & Leadership
-
Partner with data engineering, operations, and service teams to align on data quality standards and resolution workflows
-
Serve as the QA voice in sprint planning, pipeline design reviews, and platform release cycles
-
Mentor junior QA team members and help build a quality-first culture across the data organization