Senior Software QA Engineer

Crimson Education is the world’s leading college admissions consulting firm, with over 1,490 Ivy League offers and 2,410 to the US Top 15. We help ambitious students gain admission to the world’s top universities through expert-led guidance and proven, data-driven strategies. Crimson students are 7x more likely to get into the Ivy League than their peers. We were recently featured on the front page of the Wall Street Journal.

Crimson is the only college admissions consultancy that brings together:

• Former Ivy League and Top 20 admissions officers to rigorously review and refine applications

• Professors and PhD teaching fellows from leading universities to guide students through original, independent research — with pathways to selective peer-reviewed publication or conference presentation

• Past ISEF winners and judges who coach students to compete for state, national, and international science fair awards

• Capstone project mentors who help students design and scale leadership initiatives with real-world impact, measurable outcomes, and credible external validation

We’re backed by leading VC firms, including Tiger Global, Heal Partners, IceHouse Ventures, and Movac, and recently closed a USD $40M Series D funding round at a USD $640M valuation. We now operate across 21 markets worldwide, including the US, Canada, UK, Singapore, Japan, Hong Kong, Australia, and New Zealand.

The Role

What are the main responsibilities for this role?

• Design & Build: Develop, maintain, and scale robust, end-to-end automation test frameworks from scratch (UI, API, and Integration layers).

• CI/CD Integration: Seamlessly bake automated test suites into our CI/CD pipelines to ensure rapid, high-confidence deployments.

• Coverage Strategy: Define, track, and scale test coverage metrics (targeting high-purity API and critical UI path coverage) to ensure regression-free deployments.

• Performance & Scale: Conduct API performance, load, and stress testing to ensure our infrastructure keeps up with demand.

• AI/LLM Validation: Design testing strategies for non-deterministic systems. You’ll evaluate LLM outputs for accuracy, bias, safety, and hallucination rates.

• Prompt Regression Testing: Establish benchmarks to ensure that tweaking an AI prompt or upgrading a foundational model doesn't secretly break existing features.

• Data & Model Drift Monitoring: Collaborate with Data Scientists and ML Engineers to monitor model performance metrics in production.

• Edge Case Hunting: Actively simulate adversarial inputs (prompt injections, jailbreaks) to ensure our AI features remain secure and guardrailed.
  

What skills and experience are required?

• 4+ years of experience in Software QA Automation, with at least 1–2 years specifically focused on testing AI/ML-powered applications or LLM integrations.

• Strong programming proficiency in Python (highly preferred for AI workflows) or JavaScript/TypeScript.

• Hands-on experience with modern automation tools (e.g., Playwright, Cypress, Selenium).

  • Expertise in API testing tools (e.g., Postman, RestAssured, PyTest).

• Familiarity with AI evaluation frameworks (e.g., Ragas, TruLens, Deepchecks) or a proven ability to write custom evaluation scripts for probabilistic outputs.

• Basic understanding of vector databases, embeddings, and how data pipelines feed into ML models.

• Bachelor’s degree in Computer Science, Engineering, or equivalent practical experience.

Bonus Points

• Familiarity with cloud platforms (AWS, GCP, or Azure) and containerization (Docker, Kubernetes).

• A healthy obsession with AI safety and ethics.

Why work for Crimson?

• Rapidly growing start up, with a flexible working environment where you will be empowered to structure how you work

• Limitless development and exposure - our internal promotions/role changes made up 33% of all recruitment last year

• $1000 individual training budget per year, we love to ‘Level Up’ (it’s one of our core values)!

• Psychologist on staff

• Insightful fireside chats and workshops to help support our high-performing and ambitious team

• Radical Candor is a feedback approach we live by

• We’re a global player with 28 markets (and growing) across the globe. Most roles have the option to work from one of our many offices or remotely!

If you're passionate about growing in a fast-paced, collaborative environment and want to work with cutting-edge technology, then we'd love to hear from you!

Please keep an eye on your spam/junk email folder for correspondence from Team Tailor.