Offer summary

Qualifications:

Bachelor's degree in Computer Science or a related field., At least 3 years of experience in test automation, machine learning evaluation, or data analysis., Proficiency in JavaScript/TypeScript and Python, with strong scripting skills., Knowledge of AI evaluation metrics, prompt engineering, and testing non-deterministic AI systems..

Key responsibilities:

Design and maintain data-driven test automation frameworks for AI and software systems.

Develop automated evaluation pipelines and prompt-based tests for LLM and conversational AI.

Analyze logs and telemetry data to identify issues and improve user experience.

Collaborate with cross-functional teams to support continuous testing and drive testing innovations.

Job description

POSITION LOCATION: San Carlos, CA, Austin, TX, or Remote USA

POSITION SUMMARY:

We are seeking a Software Engineer in Test ( AI & Automation Focus) with experience supporting AI Driven Products to join a forwardthinking engineering team responsible for building and evaluating AIpowered systems, including a sophisticated voice assistant platform to help patients get the help they need fast. This is not a traditional SQE role this is a unique opportunity to blend test automation, machine learning evaluation, and data analysis into one impactful position.
Youll design test automation frameworks, evaluate large language model (LLM) performance, and develop custom tooling to analyze, monitor, and continuously improve product quality in both traditional software and AIdriven components.
KEY RESPONSIBILITIES:

Design, build, and maintain robust, datadriven test automation frameworks JavaScriptTypeScript and Python.

Build automated evaluation pipelines and promptbased tests for LLM and conversational AI systems.

Analyze largescale, unstructured logs and telemetry data to identify issues and user experience gaps.

Create custom scripts to support test automation, log parsing, and performance evaluation of machine learning models.

Develop visual dashboards and reports to track quality, system health, and model behavior over time.

Support test environment setup and integrate with CICD pipelines for continuous testing.

Perform root cause analysis, debugging, and reporting of product issues, collaborating closely with Developers and SQEs across the SDLC.

Drive innovation by introducing new testing methodologies, tools, and best practices for both deterministic and nondeterministic systems.

Partner with global, crossfunctional teams including ML engineers, product managers, and infrastructure leads.

QUALIFICATIONS:

BS degree in Computer Science or related technical field.

3+ years of experience in test automation, ML evaluation, data analysis, or related roles.

KNOWLEDGE, SKILLS, AND ABILITIES:

Proficient in JavaScriptTypeScript and Python; strong scripting experience, especially for automation and log parsing.

Experience with tools such as Jest, Playwright, Cypress, or Selenium for UI and API automation.

Strong knowledge of SQE principles, riskbased testing, test planning, and system integration.

Knowledge of testing approaches for AI based products, such as AI evaluations metrics and their usage.

Familiarity with LLMs, GenAI, and prompt engineering .

Comfortable working with API testing, log data analysis, and performance metrics.

Experience in evaluating and testing nondeterministic systems AIpowered projects such as conversational models or recommendation engines.

Solid understanding of CICD pipelines, cloudbased environments, and microservicesserverless architectures.

Strong communication and analytical skills; ability to clearly convey technical findings to both technical and nontechnical audiences.

Selfstarter with a growth mindset and a passion for building scalable, intelligent testing systems.

The pay range is listed and actual compensation packages are based on a wide array of factors unique to each candidate, including but not limited to skill set, years & depth of experience, certifications and specific office location. This may differ in other locations due to cost of labor considerations.
Remote USA
$94,300—$117,900 USD
OUR OPPORTUNITY

Natera™ is a global leader in cellfree DNA (cfDNA) testing, dedicated to oncology, women’s health, and organ health. Our aim is to make personalized genetic testing and diagnostics part of the standard of care to protect health and enable earlier and more targeted interventions that lead to longer, healthier lives.

The Natera team consists of highly dedicated statisticians, geneticists, doctors, laboratory scientists, business professionals, software engineers and many other professionals from worldclass institutions, who care deeply for our work and each other. When you join Natera, you’ll work hard and grow quickly. Working alongside the elite of the industry, you’ll be stretched and challenged, and take pride in being part of a company that is changing the landscape of genetic disease management.

WHAT WE OFFER

Competitive Benefits Employee benefits include comprehensive medical, dental, vision, life and disability plans for eligible employees and their dependents. Additionally, Natera employees and their immediate families receive free testing in addition to fertility care benefits. Other benefits include pregnancy and baby bonding leave, 401k benefits, commuter benefits and much more. We also offer a generous employee referral program!

For more information, visit www.natera.com .

Natera is proud to be an Equal Opportunity Employer. We are committed to ensuring a diverse and inclusive workplace environment, and welcome people of different backgrounds, experiences, abilities and perspectives. Inclusive collaboration benefits our employees, our community and our patients, and is critical to our mission of changing the management of disease worldwide.

All qualified applicants are encouraged to apply, and will be considered without regard to race, color, religion, gender, gender identity or expression, sexual orientation, national origin, genetics, age, veteran status, disability or any other legally protected status. We also consider qualified applicants regardless of criminal histories, consistent with applicable laws.

If you are based in California, we encourage you to read this important information for California residents.

Link: https:www.natera.comnoticeofdatacollectioncaliforniaresidents

Please be advised that Natera will reach out to candidates with a @natera.com email domain ONLY. Email communications from all other domain names are not from Natera or its employees and are fraudulent. Natera does not request interviews via text messages and does not ask for personal information until a candidate has engaged with the company and has spoken to a recruiter and the hiring team. Natera takes cyber crimes seriously, and will collaborate with law enforcement authorities to prosecute any related cyber crimes.

For more information:
BBB announcement on job scams
FBI Cyber Crime resource page