Spark Developer

Role: Spark Developer

Client: Client

Location: Remote

Experience in developing Spark Applications using Spark RDD API, Spark-SQL, Spark GraphX API, Spark Streaming API, Spark-Yarn, Spark MLib API, and Data frame APIs
Should have a broad knowledge of Spark Advantages, Spark Workflows, How to write Spark Jobs, Spark query tuning, and performance optimization.
Solid in Data Structures, Algorithms basics and Should have good hands-on experience with any - one programming language (Scala/ Java8 - 1st Preference OR Python - 2nd Preference OR Java - 3rd preference) and Strong investigative and problem-solving skills
Data Ingestion, Optimization Techniques, Data Transformation, and aggregation pipeline design/development knowledge is required.
Experience working on cutting-edge Big Data storage systems and technologies like Hadoop, HDFS, AWS S3, AWS Lambda, Storm/Heron, Cassandra, Apache Kafka, Solr/ElasticSearch, MongoDB, DynamoDB, Postgres, and/or MySQL, etc.

Roles & Responsibilities:
Create new, and maintain existing Scala/Spark jobs for data transformation and aggregation from simple to Complex Data transformations involving structured & unstructured data.
Produce unit tests for Spark transformations and helper methods
Develop data processing pipelines, data storage, and management architecture.
Define scalable calculation logic for interactive and batch use cases
Interact with infrastructure and data teams to produce complex analyses across data
You'll be working on a unique and challenging big data ecosystem with a focus on storage efficiency, data security, privacy, scalable and performant queries, expandability and flexibility, etc, with the goal to help better measure the quality of map data.
You will work with engineers to build a big data platform that processes and manages Exabytes of data and enables efficient access to those