AI Quality Engineer

• 1 day ago

Expedite Technology Solutions LLC

Role overview

Qualifications

QA automation testing
Playwright/Selenium, API testing (Postman, REST)
GenAI RAG validation, LLM evaluation
Prompt engineering, conversational AI testing

Responsibilities

Test Generative AI and conversational systems
Perform LLM evaluation and benchmark testing
Validate RAG (retrieval quality, relevance, grounding)
Build automation frameworks (Playwright/Selenium)

Key facts

Remote from: Anywhere
Full time
AI Operations (AI Ops) Engineer
English

Hard skills

AI Testing API Testing Test Automation Playwright (Software Testing) Selenium (Software) Prompt Engineering SQL (Programming Language) Python (Programming Language) CI/CD OpenAI Gym Azure OpenAI ChatGPT Vector Data

Other skills

Collaboration

About the company

Expedite Technology Solutions LLC

IT Services & IT Consulting

Expedite Technology Solutions has an extensive background in technology outsourcing, professional services, and staffing. We offer the expertise of a global company combined with ‘one to one’ customized service of a local business partner. Whether you want an implementation specialist for a short-term project or a software developer to add to your team, Expedite will work with you to provide the top-notch IT professionals you need. We can help you identify the right technical resources and manage the hiring process, or provide you with a complete project solution.

Company details

Company typeSME

IndustryIT Services & IT Consulting

Company size51 - 200

Links

Website LinkedIn See all jobs

Your match analysis

See how your profile stacks up against this role.

We compared the job requirements to your profile to show where you're strong and where you fall short.

Job description

Key Responsibilities

Test Generative AI and conversational systems
Perform LLM evaluation and benchmark testing
Validate RAG (retrieval quality, relevance, grounding)
Build automation frameworks (Playwright/Selenium)
Conduct API, regression, and E2E testing
Perform AI safety, red teaming, and bias validation
Support observability, monitoring, and HITL workflows
Collaborate with engineering, QA, and DevOps teams
Support CI/CD quality pipelines

Required Skills

QA & automation testing
Playwright/Selenium, API testing (Postman, REST)
GenAI & RAG validation, LLM evaluation
Prompt engineering, conversational AI testing
HITL testing, AI safety & observability
SQL, Git, Python, CI/CD concepts

Preferred

OpenAI, Azure OpenAI, ChatGPT, Claude
LangChain, CrewAI, LangGraph, MCP
Vector DBs (Pinecone, ChromaDB, Weaviate)
RAGAS, DeepEval

Python/JS, Docker, Kubernetes, GitHub Actions

About the company

Expedite Technology Solutions LLC

IT Services & IT Consulting

Company details

Company typeSME

IndustryIT Services & IT Consulting

Company size51 - 200

Links

Website LinkedIn See all jobs

Apply once. Then go straight to the hiring manager.

After you apply, unlock the direct contact details of the people who actually make the call. A quick follow-up makes you 5x more likely to land an interview.