Data Engineer (Lost)

About You:

You are passionate about data big data. You are familiar with techniques for streaming and moving data between applications. You have developed applications that efficiently and effectively move and transform data as needed.. You want to drive the accuracy, timeliness and completeness of data for all products. You want to improve processes and outcomes for data for client consumption.

Primary Accountabilities:

Technical (100%)

• Create and maintain optimal data pipeline architectures in a hybrid cloud environment
• Assemble large and complex data sets that meet both functional and non-functional business needs
• Identify, design and implement internal process improvements such as automating manual data processes, optimizing data delivery and scalability
• Build and maintain the infrastructure that is required to transfer and hold data from a wide variety of data sources
• Create analytics tools that utilize the data pipeline to provide actionable insights, operational efficiencies and other key performance metrics/indicators (KPM/KPI)
• Keep data secure throughout the pipeline in our hybrid cloud environment
• Create data tools for analytics and data science team members that will assist them in building and optimizing product portfolio
• Work with stakeholders on data-related technical issues and their data needs

Required Qualifications:

• Experience building and optimizing big data pipelines architectures and data sets.
• Working knowledge of message queuing, stream processing and highly scalable big data data stores
• You have built processes that support data transformation, data structures, metadata, dependency and workload management
• A successful history of manipulation, processing and extracting value from data sets
• Familiarity with Agile methodologies and development processes.
• You have performed root cause analysis on internal and external data and processes to answer specific business questions and identify opportunities for improvement
• Experience supporting and working with cross-functional teams in a dynamic environment
• Experience with big data tools such as Hadoop, Spark, Kafka, etc.
• Experience with public cloud technologies in Google, AWS and Azure
• Experience with relational and NoSQL databases such as SQL Server, Postgres, Cassandra, etc.
• Experience with data pipeline and workflow management tools such as Airflow
• Experience with object oriented/object function scripting languages such as Python, Java, Scala