Job description

About the Role

We're looking for a forward-thinking Automation Agent Engineer—part seasoned SDET, part prompt engineer—who’s excited to bridge the gap between classic QA automation and AI-assisted workflows. In this hybrid role, you’ll help develop and refine a mini-RAG system that transforms manual test cases into intelligent, LLM-generated test skeletons.

You'll shape how context is retrieved, how prompts are designed, and how generated test code aligns with standards for our Java-based automation stack (RestAssured, Selenium, Cucumber, Screenplay). This is a hands-on, highly collaborative role where your decisions will directly affect the quality, consistency, and scalability of our AI-enabled testing pipeline.

What You’ll Do

Design and refine prompt templates and few-shot examples to guide the LLM toward generating high-quality test code.
Collaborate on RAG logic, including chunking strategies, metadata filters, and reranker tuning for optimal context retrieval.
Help convert manual test cases into structured test skeletons, normalizing them to project standards.
Build evaluation loops to validate generated code—ensuring compliance with naming, format, assertions, lint rules, and compile/run requirements.
Review and land autoscripted code with proper guardrails, documentation, and handoff to the SDET team.
Optimize token usage and latency while maintaining test fidelity in collaboration with the AI and QA teams.

What You Bring

5+ years of experience in QA automation with strong skills in Java, RestAssured, Selenium, Cucumber, and ideally the Screenplay pattern.
Hands-on knowledge of prompt engineering, including few-shot prompting and structured output design.
Working understanding of RAG concepts (embeddings, retrievers, rerankers) and how they apply to software development workflows.
Familiarity with CI pipelines (Jenkins), Git workflows, and code review best practices.

It is an asset if you have:

Experience with LangChain, ChromaDB, or other local retrieval systems.
Understanding of LLMs like Claude 4 Sonnet and how to manage token/cost trade-offs.
Ability to build small tools such as static analyzers, linters, or rule-based code evaluators.