Logo for Nexaminds

Principal AI Quality Engineer

Nexaminds

Role overview

Qualifications

  • 7+ years of experience in Quality Engineering, Software Development, or a related technical discipline, including recent hands-on experience with AI/ML or LLM-based systems.
  • Experience designing validation or evaluation frameworks for non-deterministic or probabilistic systems such as Large Language Models (LLMs) or Machine Learning applications.
  • Strong understanding of Retrieval-Augmented Generation (RAG), agentic workflows, and LLM evaluation concepts, including hallucination detection, groundedness, relevance, and consistency.
  • Hands-on experience with Large Language Models (LLMs) and prompt engineering, including prompt design, optimization, and regression validation.

Responsibilities

  • Define and lead the AI validation strategy for LLMs, RAG systems, and AI agents.
  • Build automated validation pipelines integrated into CI/CD.
  • Establish evaluation frameworks covering correctness, relevance, groundedness, consistency, and hallucination rate.
  • Design and maintain prompt regression testing to catch silent quality degradation after model or prompt changes.

About the company

Nexaminds logo

Nexaminds

Company details

Links

LinkedIn See all jobs

Your match analysis

See how your profile stacks up against this role.

We compared the job requirements to your profile to show where you're strong and where you fall short.

Job description

Unlock Your Future with Nexaminds!

At Nexaminds, we're on a mission to redefine industries with AI. We're passionate about the limitless potential of artificial intelligence to transform businesses, streamline processes, and drive growth.

Join us on our visionary journey. We're leading the way in AI solutions, and we're committed to innovation, collaboration, and ethical practices. Become a part of our team and shape the future powered by intelligent machines. If you're driven by ambition, success, fun, and learning, Nexaminds is where you belong.

Nexaminds is looking for an AI Quality Engineer to lead the validation and quality strategy for AI-powered systems, including Large Language Models (LLMs), Retrieval-Augmented Generation (RAG) applications, and agent-based workflows. The ideal candidate has deep experience in Quality Engineering, AI evaluation, and automated validation frameworks, with a strong understanding of how to measure, monitor, and improve the reliability of non-deterministic AI systems. This role focuses on building enterprise-grade AI validation capabilities, designing automated evaluation pipelines, implementing AI guardrails, and partnering closely with engineering, AI/ML, and data governance teams to ensure AI-powered solutions are accurate, safe, and production-ready.

Location: Canada (Remote)

Qualifications we are looking for:

  • 7+ years of experience in Quality Engineering, Software Development, or a related technical discipline, including recent hands-on experience with AI/ML or LLM-based systems.
  • Experience designing validation or evaluation frameworks for non-deterministic or probabilistic systems such as Large Language Models (LLMs) or Machine Learning applications.
  • Strong understanding of Retrieval-Augmented Generation (RAG), agentic workflows, and LLM evaluation concepts, including hallucination detection, groundedness, relevance, and consistency.
  • Hands-on experience with Large Language Models (LLMs) and prompt engineering, including prompt design, optimization, and regression validation.
  • Practical experience using Claude Code, Claude SDK, or similar AI-assisted development platforms and agentic coding tools.
  • Proficiency with Node.js for building automation tools, validation pipelines, and engineering utilities.
  • Experience working with MongoDB to store, manage, and analyze evaluation results, validation data, or AI output logs.
  • Experience integrating automated validation processes into CI/CD pipelines.
  • Experience designing semantic validation approaches that evaluate meaning, relevance, and contextual accuracy beyond traditional software testing.
  • Strong understanding of AI quality metrics, confidence scoring, output consistency, and production monitoring.
  • Solid knowledge of data validation principles, including schema validation, business rule compliance, and data consistency.
  • Excellent cross-functional collaboration skills, with experience partnering across Quality Engineering, AI/ML Engineering, Software Engineering, and Data Governance teams.

Nice to have:

  • Experience with prompt regression testing tools or frameworks.
  • Familiarity with AI governance, fairness, or bias-detection practices.
  • Experience with tools such as Playwright, RestSharp, or similar UI/API test automation.
  • Prior experience standing up a validation or evaluation function from scratch.
  • Exposure to confidence scoring or guardrail systems for production AI.
  • Experience designing or orchestrating multi-agent systems in production environments

Job duties:

  • Define and lead the AI validation strategy for LLMs, RAG systems, and AI agents.
  • Build automated validation pipelines integrated into CI/CD.
  • Establish evaluation frameworks covering correctness, relevance, groundedness, consistency, and hallucination rate.
  • Design and maintain prompt regression testing to catch silent quality degradation after model or prompt changes.
  • Introduce semantic validation techniques that go beyond traditional QA (evaluating meaning, not just structure).
  • Lead production monitoring of AI output quality and define acceptable output ranges for non-deterministic systems.
  • Partner with engineering teams to validate AI-generated code, test cases, and AI-assisted decision workflows.
  • Introduce AI guardrails and confidence scoring to flag unsafe or low-confidence outputs.
  • Build validation checkpoints across multi-step agent pipelines.
  • Define and track quality metrics/KPIs (defect reduction, output consistency, prompt regression stability, adoption confidence).

What you can expect from us

Here at Nexaminds, we're not your typical workplace. We're all about creating a friendly and trusting environment where you can thrive. Why does this matter? Well, trust and openness lead to better quality, innovation, commitment to getting the job done, efficiency, and cost-effectiveness.

  • Stock options 📈
  • Remote work options 🏠
  • Flexible working hours 🕜
  • Benefits above the law
  • But it's not just about the work; it's about the people too. You'll be collaborating with some seriously awesome IT pros.
  • You'll have access to mentorship and tons of opportunities to learn and level up.

Ready to embark on this journey with us? 🚀🎉 If you're feeling the excitement, go ahead and apply!

Apply once. Then go straight to the hiring manager.

After you apply, unlock the direct contact details of the people who actually make the call. A quick follow-up makes you 5x more likely to land an interview.

MR

Marcus Rivera

Chief Revenue Officer

m.rivera@company.com
linkedin.com/in/marcusrivera
Unlocked after you apply
·

AI Operations (AI Ops) Engineer Related jobs

MexicoAI Operations (AI Ops) Engineer

Other jobs at Nexaminds

Premium

Reach out to the hiring manager directly.

Gain access to the contact details of the hiring managers who actually decide, and reach out to network with them directly. That, plus more when you upgrade:

  • Full match report with fit score and gaps
  • Career diagnostics on how recruiters read you
  • Curated company matches and warm intros
  • 48h early access to new roles

Cancel anytime.