Data Scientist – AI & Agentic Applications & Benchmarking

extra holidays - extra parental leave - work from home - fully flexible
Work set-up: 
Full Remote
Contract: 
Experience: 
Mid-level (2-5 years)
Work from: 

Offer summary

Qualifications:

Minimum of 3 years of experience in data science or ML analytics, preferably in startup or high-growth settings., Proficiency in Python, including Jupyter notebooks for analysis and visualization., Experience with evals, telemetry, A/B testing, and evaluating user-facing ML systems., Knowledge of AI/ML tools such as MLFlow, Hugging Face, and familiarity with agentic or LLM-based applications..

Key responsibilities:

  • Partner with platform teams to develop telemetry, eval frameworks, and benchmarks for agentic systems.
  • Measure AI outcomes and usage across customers and teams, and define success metrics.
  • Use Python notebooks to explore data, visualize insights, and test hypotheses.
  • Write documentation and summaries to communicate insights and outcomes.

Cloudbees logo
Cloudbees Computer Software / SaaS Scaleup http://www.CloudBees.com
501 - 1000 Employees
See all jobs

Job description

Description

About CloudBees


CloudBees is the leading software delivery platform for enterprise DevOps teams. As a high-growth startup, we empower developers to build, deploy, and manage software more efficiently. Now, we’re bringing agentic intelligence into our platform to supercharge developer workflows—and we need a data scientist who can both drive insights and tell the story behind the metrics.


The Role

CloudBees is seeking a startup-savvy Data Scientist to help define, measure, and evangelize the impact of Agentic Applications across our platform. You’ll work closely with engineers and product teams to prototype and measure AI and Agentic experiences, using evals, telemetry, and AI benchmarks to help the company drive the conversation in the market and with customers. Translating performance into clear, compelling narratives to our customers and internal teams.


As a founding member of the team, you will lead the charge as equal parts builder, evaluator, and communicator—with the technical depth to prototype in Python notebooks, Claude Code, and other tools to drive clarity to write about what matters.


Key Responsibilities
  • Partner with our platform team to develop and prototype telemetry, eval frameworks, and benchmarks for emerging agentic systems.
  • Partner with product and engineering teams to measure AI outcomes and usage across customers and teams.
  • Help define KPIs and success metrics for AI and LLM-driven features and workflows.
  • Use Python notebooks to explore data, visualize insights, and test hypotheses rapidly and share insights.
  • Tell the story behind the numbers: Write internal documentation, performance summaries, and thought leadership around outcomes.
  • Enable engineering teams to instrument, log, and evaluate agent performance effectively.
  • Stay up to date with evolving metrics and evaluation techniques in the LLM and agentic AI ecosystem.
Required Qualifications
  • 3+ years of experience in data science or ML analytics roles, ideally in startup or high-growth environments.
  • Proficiency in Python, including building and sharing analysis via Jupyter notebooks.
  • Experience working with evals, telemetry, A/B testing, and evaluating user-facing ML systems.
  • Experience with AI/ML tools such as MLFlow, Hugging Face, or other Model / LLM tools.
  • Ability to partner with technical teams to define meaningful metrics and benchmarks.
  • Clear communication skills—capable of writing about outcomes, sharing learnings, and influencing stakeholders.
  • Comfort working in fast-paced, ambiguous environments where speed and clarity matter.
Preferred Qualifications
  • Experience with agentic or LLM-based applications (e.g., evaluating AI copilots, autonomous workflows).
  • Familiarity with tools like LangSmith, OpenInference, or custom eval stacks.
  • Background in developer tools, DevOps, or platform engineering environments.
Why Join CloudBees
  • Shape the future of AI-driven DevOps with real user impact.
  • Join a nimble, passionate team at the forefront of agentic system development.
  • Work in a flexible, remote-first culture built on trust and innovation.
  • Competitive salary, startup equity, and excellent benefits.

CloudBees is proud to be an Equal Opportunity Employer. We value diverse voices, ideas, and experiences as essential to building great products.


Required profile

Experience

Level of experience: Mid-level (2-5 years)
Industry :
Computer Software / SaaS
Spoken language(s):
English
Check out the description to know which languages are mandatory.

Other Skills

  • Collaboration
  • Communication
  • Problem Solving

Data Scientist Related jobs