Big data PySpark Developer
- Visionary Innovative Technology Solutions LLC
8 Days Ago
NA
Yearly
Irving-TX
7-11 Years
Required Skills: Apache Spark, Big Data Hadoop Ecosystem, Python, SparkSQL
Job Description
- Exp. 7-11 Years needed.
- Must have good technical experience and should be able to provide technical solutions for multiple modules in parallel on need basis and bring the task to closure on time
- Unix SQL and Shell Scripting experience is a must have.
- Expertise in Designing and developing scalable Apache spark ETL based Data processing pipelines.
- Strong command line knowledge in UNIX Linux with Shell scripting using Bash Korn shell or Perl and File processing using awk scripts.
- Expertise in SQL querying and complex joins.
- Implementing comprehensive Spark based Data validation frameworks transforming large volumes of Financial data within the Project lifecycle
- Expertise with complex Data workflows with Apache Airflow managing task dependencies SLAs etc to ensure timely data delivery and corresponding automated validation controls
- Strong Analytical skills and expertise on Spark SQL for Data analysis and validation ensuring the delivery of clean query ready datasets for business consumption
- Expertise in Data quality checks and monitoring
- Quality Engineering team where 70percent of effort will be for developing automation frameworks for testing Remaining 30percent effort will be on manual testing until its fully automated
- Hands-on with Automation Framework Design for ETL and API
- SME in Data Analysis Database testing Messaging queues
- Experience with coding standards code reviews source management build processes CICD pipeline.
