Logo for mpathic

Machine Learning Engineer, Evaluation

Role overview

Summary

This role focuses on building and improving ML-powered products at mpathic, particularly in evaluating their quality and reliability. You will work across the ML lifecycle, contributing to production systems and helping the team make informed decisions about models and data.

Qualifications

  • Practical experience developing and shipping ML systems, typically 2–5 years of relevant work or equivalent.
  • Strong Python skills and familiarity with common ML frameworks.
  • Solid ML fundamentals, including training, validation, generalization, and common sources of model error.
  • Experience with evaluation metrics, benchmark design, and systematic error analysis.
  • An understanding of statistical inference, sampling, confidence intervals, and experimental design.

Responsibilities

  • Build, deploy, and maintain ML models and services in collaboration with engineering and product teams.
  • Design evaluation approaches that reflect product needs and real-world operating conditions.
  • Create and maintain evaluation datasets, benchmarks, and automated regression tests that measure safety, empathy, and risk.
  • Communicate findings, limitations, and recommendations clearly to clinicians, product leads, and customers.

Key facts

Hard skills

Key skills for this role

Other skills

  • Communication
  • Teamwork

About the company

mpathic logo

mpathic

  • AI & Machine Learning
  • Startup
  • 11–50
mpathic is keeping humans safe in the AI era through technology-enabled, expert-led human data and evaluation services. We work with frontier AI builders to prevent harmful or unwanted model behaviors, spanning use cases from user wellbeing and mental health to financial risk and customer support, across the full AI model lifecycle. mpathic Experts create multi-modal ground-truth datasets, trajectories, annotations and role plays for use in red teaming and benchmarking. Experts have specialization in behavioral analysis, conversational design, and high-risk/high-accuracy areas. We partner with Responsible AI and Trust & Safety teams to support global deployment, policy enforcement, and ongoing risk mitigation to strengthens both model quality and real-world safety. Learn more at mpathic.ai

Your match analysis

See how your profile stacks up against this role.

We compared the job requirements to your profile to show where you're strong and where you fall short.

Job description

About mpathic

mpathic builds AI that understands clinical and human conversations: analyzing therapy and clinical-trial sessions, producing expert-labeled data for AI builders, and publishing open safety benchmarks.

About the role

We’re looking for a Machine Learning Engineer to build and improve ML-powered products, with a particular focus on evaluating their quality and reliability. You’ll work across the ML lifecycle, combining hands-on engineering with rigorous experimentation to understand how systems perform and where they need to improve.

This is a generalist ML engineering role with an evaluation focus. You’ll contribute to production systems while helping the team make sound decisions about models, data, and product changes.

What you’ll do

  • Build, deploy, and maintain ML models and services in collaboration with engineering and product teams.
  • Design evaluation approaches that reflect product needs and real-world operating conditions.
  • Create and maintain evaluation datasets, benchmarks, and automated regression tests that measure safety, empathy, and risk.
  • Select appropriate metrics, analyze performance across meaningful data segments, and investigate failure modes.
  • Design experiments and use statistical methods to judge whether an improvement is real.
  • Connect offline evaluation results with what customers see in production.
  • Develop tools that make evaluation repeatable and useful throughout development and release workflows.
  • Communicate findings, limitations, and recommendations clearly to clinicians, product leads, and customers.

What you’ll bring

  • Practical experience developing and shipping ML systems, typically 2–5 years of relevant work or equivalent.
  • Strong Python skills and familiarity with common ML frameworks.
  • Solid ML fundamentals, including training, validation, generalization, and common sources of model error.
  • Experience with evaluation metrics, benchmark design, and systematic error analysis.
  • An understanding of statistical inference, sampling, confidence intervals, and experimental design.
  • Experience identifying data leakage, evaluation bias, and gaps in test coverage.
  • Good software engineering practices, including testing, version control, and code review.
  • Skill at turning ambiguous product goals into measurable quality criteria.

Helpful additional experience

  • A/B testing and production performance monitoring.
  • Human evaluation, annotation guidelines, and inter-rater agreement.
  • Speech or conversational data: transcripts, diarization and multi-speaker setups.
  • Evaluating ranking, retrieval, recommendation, or generative AI systems.
  • Robustness testing, calibration, fairness analysis, or distribution shift detection.
  • Building evaluation tooling and integrating quality checks into CI/CD workflows.
  • Clear, direct communication, both written and verbal.
  • Startups and cross-functional teams.

Related jobs

See all Machine Learning Engineer jobs

Other jobs at mpathic