Offer summary
Qualifications:
7-10 years experience in Scala, Java, Expertise in AWS EMR, Spark.Key responsabilities:
- Design and develop ETL pipeline
- Implement data quality checks
- Identify machine learning application areas
Match score not available
Priority access
Match score not available
Big Data Engineer
72% Flex
Remote:
Full Remote
Experience:
Expert & Leadership (>10 years)
Job description
Your missions
Senior Big Data Developer/Engineer
100% Remote only EST candidates
12 months contract
Must Have
- Strong experience with Scala, Spark, EMR
- Lead/Senior level Data Engineering with Spark Scala AWS EMR and lambda.
- 7-10 years in total would be ideal experience.
Top Skills' Details
This person will not be leading the team or have any direct reports but will be a senior developer on the team who can provide technical mentorship to other engineers.
- At least 7 years of experience programming in Scala, Java
- Strong experience in big data technologies like AWS EMR and Apache Spark
- Strong experience with serverless technologies like AWS Dynamo DB and AWS Lambda
- Chosen database is AWS Aurora
- Technical experience in all of the areas listed below:
- Experience working with JSON files as data will be coming in as JSON files
- Ability to write complex SQL queries
- Strong experience in performance tuning and optimization
- Strong unit testing using JUnit or ScalaTest is minimum expectation, data testing experience would be great
- Git/Maven/Gradle
Tech Stack
- Scala is main programming language for the team
- Aurora is database
- SQL is backend database
- ETL process is Scala/Spark on EMR clusters
- Code reviews are large aspect of team culture
- Agile environment with 2 week sprints
Job Responsibilities
- Understand complex business requirements
- Design and develop ETL pipeline for collecting, validating and transforming data according to the specification
- Develop automated unit tests, functional tests and performance tests.
- Maintain optimal data pipeline architecture
- Design ETL jobs for optimal execution in AWS cloud environment
- Reduce processing time and cost of ETL workloads
- Lead peer reviews and design/code review meetings
- Provide support for production support operations team
- Implement data quality checks.
- Identify areas where machine learning can be used to identify data anomalies
Required profile
Experience
Level of experience: Expert & Leadership (>10 years)
Spoken language(s):
English
Check out the description to know which languages are mandatory. Soft Skills
- Mentorship
Share
Go Premium: Access the World's Largest Selection of Remote Jobs!
- Largest Inventory: Dive into the world's largest remote job inventory. More than half of these opportunities can't be found on standard platforms.
- Personalized Matches: Our AI-driven algorithms ensure you find job listings perfectly matched to your skills and preferences.
- Application fast-lane: Discover positions where you rank in the TOP 5% of applicants, and get personally introduced to recruiters with Jobgether.
- Try out our Premium Benefits with a 7-Day FREE TRIAL.
No obligations. Cancel anytime.