Experience with big data tools: Hadoop, Spark, Kafka
Experience with data pipeline and workflow management tools: Airflow
Experience with AWS cloud services: EC2, EMR, AWS Glue
Experience with programming language: Python
Requirements:
Build data pipelines from ground up and assemble large, complex data sets to meet business requirements
Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, and redesigning infrastructure for scalability
Build ETL pipelines to extract, transform, and load data from diverse sources using AWS Glue and EMR
Develop RESTful APIs and analytics capabilities to derive actionable insights and enable third-party integrations
Job description
Position: Senior Data Engineer
Location: Remote Duration: Contract
Rate: DOE
Job Description : Looking for data engineers who can build data pipelines from grounds up. The candidate should be able to assemble large, complex data sets that meet functional / non-functional business requirements.Identify, design, and implement internal process improvements: automating manual processes, optimizing data delivery, re-designing infrastructure for greater scalability, etc.
Build pipelines to extraction, transformation, and loading of data from a wide variety of data sources using AWS Glue / EMR.
Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics.
Develop RestFul API to integrate with third party APIs using a framework.Experience building and optimizing big data data pipelines, architectures and data sets.
Strong analytic skills related to working with unstructured datasets.
Working knowledge of message queuing, stream processing, and highly scalable big data data stores.
Experience with big data tools: Hadoop, Spark, Kafka, etc.
Experience with data pipeline and workflow management tools: Airflow
Experience with AWS cloud services: EC2, EMR, AWS Glue
Experience with stream-processing systems: Storm, Spark-Streaming, etc.
Experience with programming language: Python