Big Data Engineer
- Excelon Solutions
2 Days Ago
NA
C2C
Remote
8-10 Years
Required Skills: AWS, Kubernetes, Spark, Python, SQL,
Job Description
Big Data Engineer (AWS EKS/Kubernetes SME)
This is NOT a standard Data Engineer role. The hiring manager needs an SME who has implemented and supported Spark workloads on EKS/EMR on EKS in large-scale environments. Strong AWS, Kubernetes, Spark, Python, SQL, and Big Data experience required.
AWS/K8s certifications highly preferred. Please send qualified resumes with rate, location, availability, and cert details.
Required Experience
✅ Deep expertise with AWS EKS (Elastic Kubernetes Service)
✅ Experience running and optimizing Apache Spark workloads on Kubernetes
✅ Hands-on experience with EMR on EKS
✅ Strong understanding of:
- Kubernetes architecture
- Pods, deployments, namespaces, services
- Resource allocation and autoscaling
- Helm
- Kubernetes networking and security
- Troubleshooting cluster and workload performance issues
✅ Big Data technologies including:
- Spark
- Hadoop
- Hive
- Trino
✅ AWS Services:
- EKS
- EMR
- S3
- Glue
- Athena
- Lambda
✅ Strong coding background in:
- Python
- SQL
- Scala (preferred)
✅ Experience working with petabyte-scale data processing environments
Nice to Have
- Terraform or CloudFormation
- CI/CD (Jenkins, GitLab, GitHub Actions, ArgoCD)
- Docker
- Prometheus/Grafana
- Service Mesh technologies
- Financial Services experience
- GenAI tools (Copilot, ChatGPT, Claude, Amazon Q)
MUST HAVE
The manager is specifically looking for candidates who have:
- Proven AWS EKS expertise
- Spark-on-Kubernetes experience
- Large-scale Big Data engineering experience
- Relevant AWS and/or Kubernetes certifications
