Logo for Softgic

Agent Quality / Evals Engineer 1754

Softgic

Role overview

Qualifications

  • Experience evaluating ML, LLM, or non-deterministic systems.
  • Strong test and benchmark design capability.
  • Comfort working with noisy metrics, thresholds, and probabilistic behavior.
  • Good scripting and automation skills.

Responsibilities

  • Build and maintain the MVP eval harness: golden tasks, exception tasks, scorecard metrics, and regression packs.
  • Wire evals into CI so quality regressions fail builds and releases.
  • Define and maintain release-gate thresholds with Product and the Tech Lead.
  • Lay the path for later adversarial and drift-testing expansion without overbuilding MVP scope.

Key facts

Hard skills

Benchmarking Scripting Test Design Business Metrics AI Testing

Other skills

  • Quality Assurance
  • Problem Solving
  • Teamwork

About the company

Softgic logo

Softgic

We are a young and growing company, with operations in Medellin and Bogota, focused on the generation of technological solutions in synergy with our customers and our team so that these solutions add value within their organizations and their business processes.

Company details

Company size51 - 200

Links

Website LinkedIn See all jobs

Your match analysis

See how your profile stacks up against this role.

We compared the job requirements to your profile to show where you're strong and where you fall short.

Job description

This is a remote position.

Owns the eval harness and quality gate from the beginning. This role replaces the old late-stage “Evals Specialist” model with a standing owner for measurable agent quality.

Key Responsibilities

• Build and maintain the MVP eval harness: golden tasks, exception tasks, scorecard metrics, and regression packs.
• Wire evals into CI so quality regressions fail builds and releases.
• Define and maintain release-gate thresholds with Product and the Tech Lead.
• Lay the path for later adversarial and drift-testing expansion without overbuilding MVP scope.


Requirements

Must-Have Qualifications

• Experience evaluating ML, LLM, or non-deterministic systems.
• Strong test and benchmark design capability.
• Comfort working with noisy metrics, thresholds, and probabilistic behavior.
• Good scripting and automation skills.

AI-First Expectations

• Uses AI to generate candidate eval cases and failure hypotheses, but never confuses generated tests with validated quality.
• Approaches AI quality as an operating system, not a QA afterthought.

What Success Looks Like in the First 90 Days

• The first reference agent has a published scorecard and gated eval path. • Golden and exception tests run automatically. • The team can explain what “good enough to ship” means in measurable terms.

Apply once. Then go straight to the hiring manager.

After you apply, unlock the direct contact details of the people who actually make the call. A quick follow-up makes you 5x more likely to land an interview.

MR

Marcus Rivera

Chief Revenue Officer

m.rivera@company.com
linkedin.com/in/marcusrivera
Unlocked after you apply
·

Quality Improvement Engineer Related jobs

WorldwideQuality Improvement Engineer

Other jobs at Softgic

Premium

Reach out to the hiring manager directly.

Gain access to the contact details of the hiring managers who actually decide, and reach out to network with them directly. That, plus more when you upgrade:

  • Full match report with fit score and gaps
  • Career diagnostics on how recruiters read you
  • Curated company matches and warm intros
  • 48h early access to new roles

Cancel anytime.