GCP Data Engineer
- Fixity Technologies
6 Days Ago
NA
Yearly
Charlotte NC-NC
8-12 Years
Required Skills: GCP , BigQuery Airflow , Cloud Composer , DataProc , Apache Kafka , Apache Spark , Hadoop , Hive , Python , Shell Scripting , SQL , Data Modeling , Data Pipelines , Jenkins , Git , GitHub , CI/CD , Automated Testing , Object-Oriented Programming , Design Patterns , Distributed Systems , Agile, Scrum
Job Description
We are seeking an experienced GCP Data Engineer with 8+ years of hands-on experience in software development and Big Data & Analytics solutions. The ideal candidate will have strong expertise in Google Cloud Platform, BigQuery, Airflow, DataProc, Pub/Sub, Hadoop, Hive, Spark, Python, Shell Scripting, and SQL.
The candidate will be responsible for designing, developing, optimizing, and maintaining scalable data pipelines and distributed data processing solutions using modern cloud and Big Data technologies.
Key Responsibilities:
- Design, develop, and optimize scalable data pipelines for large-scale data processing and transformation.
- Develop cloud-based data engineering solutions using GCP services, including BigQuery, DataProc, Airflow, and Pub/Sub.
- Design and optimize data models, databases, and SQL queries for performance and scalability.
- Build batch and real-time/event-driven data processing pipelines using Apache Kafka and/or GCP Pub/Sub.
- Develop data processing solutions using Apache Spark, Hadoop, and Hive.
- Write efficient and maintainable code using Python and Shell Scripting.
- Apply strong Object-Oriented Programming concepts and software design patterns.
- Design and develop solutions for distributed and multi-tiered systems.
- Build and maintain CI/CD pipelines using Jenkins, Git, and GitHub.
- Implement automated testing frameworks and quality practices for data engineering solutions.
- Follow software development lifecycle processes and Agile/Scrum methodologies.
- Collaborate effectively with product teams, architects, developers, and other cross-functional teams.
- Analyze complex data engineering problems and develop effective, scalable solutions.
- Identify opportunities for automation, process improvement, and continuous improvement.
- Learn and adopt enterprise frameworks and emerging technologies to build modern Big Data solutions.
Required Skills:
- 8+ years of experience in software development and/or Big Data Engineering.
- Strong hands-on experience with Google Cloud Platform (GCP).
- Strong experience with BigQuery.
- Experience with Apache Airflow / Cloud Composer.
- Experience with Google Cloud DataProc.
- Experience with GCP Pub/Sub and/or Apache Kafka.
- Strong knowledge of Apache Spark, Hadoop, and Hive.
- Strong programming experience in Python.
- Experience with Shell Scripting.
- Strong SQL and relational database experience.
- Experience designing and optimizing data models and data pipelines.
- Experience with Jenkins, Git, and GitHub.
- Knowledge of CI/CD pipelines and automated testing frameworks.
- Strong understanding of distributed systems, algorithms, and software design patterns.
- Good understanding of Object-Oriented Programming concepts.
- Experience working in Agile/Scrum environments and understanding of SDLC processes.
- Strong analytical, communication, and problem-solving skills.
Preferred Qualifications:
- Google Cloud Professional Data Engineer Certification is a plus.
- Experience with enterprise Big Data frameworks and architecture.
- Knowledge of infrastructure automation and configuration management.
- Experience with continuous integration and deployment practices.
- Strong curiosity and willingness to learn new technologies.
