The Data Engineer is responsible for building and maintaining systems that improve data reliability, quality, and accessibility. This role focuses on developing software and data pipelines that extract, transform, and load (ETL) large volumes of data from multiple sources into consistent, machine-readable formats. Data Engineers design and test architectures that support data extraction and transformation for predictive and prescriptive analytics. Working within an agile Scrum environment, the Data Engineer participates in two-week sprint cycles, contributing to sprint planning, development, and review processes to continuously improve data infrastructure and product capabilities.
Major Responsibilities:
Data Transformation: Collect, manage, and transform large-scale raw data from multiple sources into structured, reliable, and usable formats that meet business and analytical requirements.
Data Tool Creation: Design, develop, and maintain data architectures, pipelines, and applications using appropriate tools and frameworks to support scalable data processing and product development.
ETL Development: Build and maintain robust extract, transform, and load (ETL) processes that ensure data accuracy, efficiency, and availability across systems.
Root Cause Analysis & Continuous Improvement: Perform root cause analysis to identify data issues, system inefficiencies, or process gaps, and implement improvements that enhance data quality and operational performance.
Agile Collaboration: Work in an agile Scrum environment by contributing to sprint planning, development tasks, testing, and sprint reviews to deliver reliable and scalable data solutions
Education and Experience:
Bachelor’s in Computer Science, Mathematics, Statistics, or related field (or equivalent experience)
2+ years in data engineering, data analysis, or similar roles
Experience with ETL, data warehousing, and scalable data systems.
Solid understanding of statistical methods and data analysis
Strong communication skills with ability to work independently and in teams
Proficiency in Python and SQL; familiarity with Java, C/C++, or similar
Strong software engineering, programming, and QA skills
Experience with Linux/Unix (POSIX systems) and Git
Preferred Qualifications
Experience with Docker, Terraform, and AWS (Batch, EMR/Spark, ecosystem tools)
Familiarity with GitLab and Scala
Lightcast is a global leader in labor market insights with our headquarters located in Moscow, ID (US) and other offices in the United Kingdom, Europe, and India. We work with partners across six continents to help drive economic prosperity and mobility by providing the insights needed to build and develop our people, our institutions and companies, and our communities.
Lightcast is proud to be an equal opportunity workplace and does not discriminate on the basis of federally protected statuses. Lightcast has always been, and always will be, committed to diversity of thought and unique perspectives. We seek dynamic professionals from all backgrounds to join our teams, and we encourage our employees to bring their authentic, original, and best selves to work.
