Sr. Big Data Engineer
- Fuge Technologies Inc.
10 Days Ago
NA
NA
Weehawken-NJ
11-18 Years
Required Skills: java, azure, databricks
Job Description
Sr. Big Data Engineer at Weehawken, NJ
Client: UBS
Our Challenge
Client: UBS
Our Challenge
- As a Big Data Engineer, candidate will be instrumental in developing and optimizing big data technologies such as Apache Spark and Azure Databricks.
- Candidate will create robust data pipelines, ensure high data quality, and collaborate closely with cross-functional teams to implement scalable cloud-native solutions on major cloud providers, predominantly Microsoft Azure.
The Role
Responsibilities:
- Develop, optimize, and maintain big data pipelines using Apache Spark, Azure Databricks, and related tools.
- Write and deploy complex production systems in Scala, Java, or Python, ensuring high performance and reliability.
- Leverage Terraform for automating cloud infrastructure deployment on Azure or other major cloud providers.
- Design and implement CI/CD pipelines for data workflows using modern tools like GitLab, promoting automation and continuous improvement.
- Apply best practices for testing, instrumentation, observability, and alerting to maintain data pipeline health and performance.
- Contribute to system architecture and low-level design, ensuring modularity, scalability, and security.
- Understand and implement data models, data structures, and algorithms for efficient data processing.
- Use containerization technologies such as Docker and Kubernetes for development, build, and runtime environments.
- Work within Agile development teams, participating in planning, daily stand-ups, and iterative releases.
- Collaborate effectively in a global team, influencing key architectural decisions and sharing best practices.
Requirements:
- 10+ Years of experience is required.
- Proven experience developing big data solutions with Apache Spark and Azure Databricks.
- Hands-on experience with Terraform for infrastructure as code (IaC) on major cloud platforms, ideally Azure.
- Programming expertise in Scala, Java, or Python for complex production systems.
- Extensive Python experience, especially in data engineering contexts.
- Background in platform engineering roles on cloud platforms, with strong knowledge of Azure services.
- Practical knowledge of testing frameworks, instrumentation, observability, and alerting tools.
- Experience building and maintaining CI/CD pipelines with modern cloud-friendly systems like GitL
