Logo for EPAM Systems

Lead Data Engineer (PySpark, statistics, MLflow)

Role overview

Summary

This role focuses on addressing complex data engineering challenges, including migrating legacy codebases and building production-grade pipelines. You will work independently on large-scale PySpark systems and apply statistical methods to analyze sensor data, contributing to the management of machine learning models using MLflow.

Qualifications

  • 5+ years of experience in data engineering with deep knowledge of PySpark and pandas.
  • Proven experience migrating pandas, Koalas or pandas-on-Spark code to native PySpark, including Window functions.
  • Expertise in applied statistics, including outlier detection and event detection in sensor time series.
  • Proficiency in MLflow for experiment tracking, Model Registry and versioning, with skills in packaging existing models including explainability techniques like SHAP.
  • Preferred: English proficiency at B2 level or higher.

Responsibilities

  • Migrate existing pandas, Koalas or pandas-on-Spark code to native PySpark implementations
  • Decompose and refactor large legacy notebooks without relying on existing documentation
  • Apply statistical techniques to detect outliers and events in sensor time series data
  • Package and manage machine learning models using MLflow, including experiment tracking, Model Registry and versioning

Key facts

Hard skills

Key skills for this role

Other skills

  • Problem Solving

About the company

EPAM Systems logo

EPAM Systems

  • IT Services & Consulting
  • XLarge
  • 10,001+
Since 1993, EPAM Systems, Inc. (NYSE: EPAM) has used its software engineering expertise to become a leading global provider of digital engineering, cloud and AI-enabled transformation services, and a leading business and experience consulting partner for global enterprises and ambitious startups. We address our clients’ transformation challenges by fusing EPAM Continuum’s integrated strategy, experience and technology consulting with our 30+ years of engineering execution to speed our clients’ time to market and drive greater value from their innovations and digital investments. We make GenAI real with our AI LLM orchestration, testing and engineering solutions, EPAM DIAL, EPAM EliteA™ and EPAM AI/RUN™, respectively. We deliver globally, but engage locally with our expert teams of consultants, architects, designers and engineers, making the future real for our clients, our partners and our people around the world. We believe the right solutions are the ones that improve people’s lives and fuel competitive advantage for our clients across diverse industries. Our thinking comes to life in the experiences, products and platforms we design and bring to market. Added to the S&P 500 and the Forbes Global 2000 in 2021 and recognized by Glassdoor and Newsweek as a Top 100 Best Workplace, our multidisciplinary teams serve customers across six continents. We are proud to be among the top 15 companies in Information Technology Services in the Fortune 1000 and to be recognized as a leader in the IDC MarketScapes for Worldwide Experience Build Services, Worldwide Experience Design Services and Worldwide Software Engineering Services. Learn more at www.epam.com.

Your match analysis

See how your profile stacks up against this role.

We compared the job requirements to your profile to show where you're strong and where you fall short.

Job description

We are looking for a Lead Data Engineer to join our team and take ownership of complex data engineering challenges, from migrating legacy codebases to building robust, production-grade pipelines. In this role, you will work independently on large-scale PySpark systems, apply statistical methods to real-world sensor data, and leverage MLflow to manage the full lifecycle of machine learning models. Responsibilities Migrate existing pandas, Koalas or pandas-on-Spark code to native PySpark implementations Decompose and refactor large legacy notebooks without relying on existing documentation Apply statistical techniques to detect outliers and events in sensor time series data Package and manage machine learning models using MLflow, including experiment tracking, Model Registry and versioning Implement model explainability techniques such as SHAP for existing models Develop unit and regression tests for PySpark pipelines using pytest and tools like assertDataFrameEqual or chispa Work independently on data engineering projects with minimal oversight Integrate AI coding assistants into daily workflows and critically review their output Utilize Window functions to optimize data transformation logic Requirements 5+ years of experience in data engineering with deep knowledge of PySpark and pandas Proven experience migrating pandas, Koalas or pandas-on-Spark code to native PySpark, including Window functions Capability to read and decompose large legacy notebooks without documentation, and to work independently on them Expertise in applied statistics, including outlier detection and event detection in sensor time series Proficiency in MLflow for experiment tracking, Model Registry and versioning, with skills in packaging existing models including explainability techniques like SHAP Skills in writing unit and regression tests for PySpark using pytest and assertDataFrameEqual or chispa Familiarity with using AI coding assistants in daily work and reviewing their output critically English proficiency at B2 level or higher Nice to have Knowledge of incremental processing on Delta, including MERGE and Change Data Feed, and Databricks job triggers Background in working with sensor or IoT data Databricks Machine Learning Professional certification German proficiency

Related jobs

See all Data Engineer jobs

Other jobs at EPAM Systems