Lead Data Engineer

Position Summary: 

As members of the Application Services team, Data engineers are responsible for creating and maintaining data pipelines, assembling complex datasets, and extracting, transforming, and loading data from a variety of data sources. Data engineers consult with development squads and data scientists to deliver product features and with cloud database administrators to resolve data related technical issues. Data engineers also identify and implement process improvements for greater reliability and performance.

Performs work with a high degree of latitude. Handles the most complex issues. Possesses expert knowledge of subject matter. Provides leadership, coaching, and/or mentoring to subordinate a group. May act as technical lead or first-level supervisor. Bachelor's degree or equivalent professional experience required; Computer Science or Engineering preferred. Minimum of 8 or more years of experience.

Data engineers work as members of a Center of Excellence and participate in projects to ensure that Truckstop has hygienic, accurate, data available at the point of need.

Data Engineers may troubleshoot production issues and respond to incidents.

Essential Job Functions: 

• Create, automate, and maintain data pipelines. Assemble complex data sets that meet business requirements.
• Extract, transform, and load data from a variety of data sources using SQL and big data tools and technologies.
• Identify, design, and implement internal process improvement including automating manual processes, optimizing data delivery, re-designing infrastructure for greater reliability and performance.
• Write and optimize queries, write lambda functions, perform source to target mapping and data modeling.
• Build analytics tools that utilize the data pipeline to provide actionable insights into customer acquisition, operational efficiency and other key business performance metrics.
• Work as a member of the data engineering chapter on a development squad to deliver product features and resolve data related technical issues.
• Work as a member of the data management center of excellence on data quality projects.
• Work with information security to keep our data secure.
• Work with data architects, database administrators, and data engineers to continually improve our data systems.
• Work with IT operations to resolve data related technical issues and respond to data related major incidents.
• Create data tools to assist data analysts and data scientists in building and optimizing our product into an innovative industry leader.
• Troubleshoot operations and database issues for internal and external customers.
• Perform incident resolution and root cause analysis of critical outages. Implement solutions to systematic failures. Provide on-call support, including after hours.
• Assist with documentation of the environments that support our products.
• Other duties as assigned.

Position Requirements: 

• Bachelor's degree or equivalent professional experience required; Computer Science or Engineering preferred.
• Strong experience in SQL and Python.
• Experience with relational SQL and NoSQL Databases; MS SQL, PostgreSQL, Snowflake, MongoDB, MySQL, MariaDB, Elastic stack.
• Experience building and optimizing reliable, idempotent data pipelines (streaming or batch) using ETL tools such as Matillion, Apache NiFi, Kafka and Spark.
• A successful history applying software engineering principles to build data infrastructure tools/libraries to automate and scale data pipelines.
• Understanding of applying logging and metrics to monitor and detect data pipeline performance issues.
• Solid understanding of Data warehousing theory and concepts. 
• Working knowledge of data modeling and source to target mapping with a demonstrated understanding of data schema design and indexes, when to apply fact-based vs. time-based vs. relational and when to apply NoSQL approaches such as key-value, document, and column data stores.
• Understanding of Statistical and Machine Learning (ML) terminology with experience deploying and monitoring ML-models in a production environment.
• Experience working in a modern cloud-based environment a plus; Azure, AWS, Docker, Kubernetes. Data Warehouse
• Experience with Linux.
• Familiarity with Kubernetes, Lambda functions, Cloud Watch, SNS, S3, Glacier.
• Minimum of 8 or more years of experience.