Data Engineer
- Apetan Consulting LLC
2 Days Ago
NA
C2C
Phoenix-AZ
7-10 Years
Required Skills: Azure DevOps, ETL, ELT, Snowflake, SQL
Job Description
Role Overview
The Data Engineer is a hands-on delivery role responsible for developing, testing, implementing, and supporting enterprise data integration solutions using Informatica IDMC and Snowflake. This position operates as a staff augmentation resource and works under the direction of company leadership and architectural standards to deliver data ingestion, transformation, data quality, and integration capabilities that support enterprise analytics, reporting, and operational workloads. The role is expected to execute assigned deliverables, participate in project initiatives, and support operational objectives while adhering to established governance, security, and development standards.
This role is primarily responsible for hands-on development, administration, and support activities and is not intended to serve as an enterprise architecture, strategy, or advisory role.
Key Responsibilities
ETL/ELT Development
- Design, build, and maintain data pipelines using Informatica IDMC, including mappings, mapplets, transformations, and task flows.
- Develop and optimize SQL‑based transformations in Snowflake across staging, integration, and curated layers.
- Implement reusable frameworks for error handling, data cleansing, parameterization, and pipeline orchestration.
Data Quality & Testing
- Perform unit, integration, and regression testing for new or modified data pipelines.
- Execute post‑load reconciliation, validation rules, completeness checks, and anomaly detection.
- Implement automated data quality workflows using IDMC and/or Snowflake Tasks & Streams.
Environment & Release Management
- Support migration of Informatica IDMC artifacts between Dev and Prod environments.
- Ensure IDMC pipelines, connections, and Secure Agent configurations are ready for deployments.
Operational Responsibility
- Monitor pipeline executions, address load failures, and drive resolution of data quality and processing issues.
- Diagnose root causes related to schema drift, source system changes, or unexpected data patterns.
- Collaborate with Platform Administrators to support reliable execution of scheduled workflows in JAMS.
Governance, Documentation & Metadata
- Develop and maintain mapping documents, data flow diagrams, and end‑to‑end lineage documentation.
- Ensure all datasets meet governance requirements, including glossary alignment, classification, and usage rules.
- Maintain metadata accuracy within governance tools and Snowflake information schema.
Security & Compliance
- Ensure pipelines follow data handling standards, including PII protections, masking, and role‑based access expectations.
- Participate in audit support, access reviews, and change control processes as required.
- Work with Data Warehouse Engineers to align models, transformations, and performance expectations.
- Work closely with platform engineering teams to ensure Secure Agents are running, monitored, and capable of supporting required pipeline workloads.
- Collaborate with Analytics and BI teams to deliver accurate, analytics-ready datasets.
- Collaborate with the Data Governance team to support data quality, consistency, and adherence to enterprise standards for cleanliness, accuracy, and reliability.
- Collaborate with IT Security and Risk teams to implement approved security controls and remediation activities in accordance with enterprise security standards and risk management requirements.
Skills & Tools
- Informatica IDMC / IICS & Cloud Data Integration (CDI): 5+ years of experience designing, developing, and supporting enterprise-scale data integration solutions using Informatica Intelligent Data Management Cloud (IDMC/IICS), including mappings, mapplets, taskflows, parameterization, reusable frameworks, batch and incremental data ingestion, transformation logic, and ETL/ELT development.
- Snowflake SQL: 5+ years of advanced SQL development experience, including complex query optimization, performance tuning, ELT development, and transformation implementation within Snowflake environments.
- Informatica Data Quality (CDQ): Experience developing data profiling, validation rules, reconciliation processes, data quality monitoring, exception handling, and automated quality controls.
- Informatica Cloud Data Governance & Catalog (CDGC): Working knowledge of metadata management, data lineage, business glossary alignment, cataloging, and governance processes. Experience supporting regulatory compliance initiatives is preferred.
- Production Support & Performance Optimization: Experience monitoring pipeline executions, troubleshooting failures, resolving data integration issues, optimizing ETL/ELT performance, and supporting enterprise production environments.
- Git & Azure DevOps: 5+ years of experience utilizing Git-based source control and Azure DevOps for collaborative development, branching strategies, pull requests, code reviews, work item management, release management, and deployment automation across data integration and data warehouse solutions.
- CI/CD Pipelines & Deployment Automation: 5+ years of experience designing, implementing, and supporting CI/CD pipelines for Informatica IDMC/IICS and Snowflake deployments, including automated testing, code promotion, environment migration, release management, and deployment automation using Azure DevOps, GitHub Actions, GitLab CI/CD, or equivalent platforms.
- Financial Services Experience: Preferred.
