AI Evaluation Engineer - Business & Operations Domain

Gramian Consultancy is a boutique consultancy specializing in IT professional services and engineering talent solutions. With a strong background in software engineering and leadership, we help companies build high-performing teams by matching them with professionals who truly fit their needs.

Role Overview

We are looking for highly analytical business and operations professionals to contribute to advanced AI evaluation projects focused on realistic operational workflows, business processes, and multi-step decision-making scenarios. In this role, you will help design and evaluate complex tasks that test how effectively AI systems handle real-world operational challenges across business environments.

The ideal candidate has hands-on experience working with business operations, analytics, workflow automation, reporting, CRM systems, financial processes, or cross-functional operational workflows. You will contribute realistic scenarios involving process optimization, business reasoning, operational failures, reporting workflows, and structured decision-making.

This role is particularly well suited for professionals with backgrounds in business operations, management consulting, business analytics, finance operations, CRM and sales operations, supply chain and logistics, operational excellence, process automation, reporting and BI, strategy consulting, or workflow optimization.

CONTRACT: Contractor assignment (5 weeks)

COMMITMENT: Full-time (40h/week) or Part-time (20h/week) with minimum 4h PST overlap

LOCATION: Remote — Bangladesh, Brazil, Colombia, Egypt, Ghana, India, Pakistan, Indonesia, Kenya, Nigeria, Turkey, Vietnam

PROCESS: One technical assessment/interview (~45 min)

Responsibilities:

• Design realistic business and operational workflow scenarios for AI evaluation systems
• Create structured tasks involving analytics, reporting, operational reasoning, and process optimization
• Develop clear task specifications, expected outcomes, and validation logic
• Identify operational edge cases, bottlenecks, and workflow failure scenarios
• Evaluate AI-generated outputs for reasoning quality, usefulness, and accuracy
• Contribute expertise across business operations, analytics, automation, or operational systems
• Review and improve workflow complexity, clarity, and evaluation quality
• Collaborate with reviewers and researchers to refine AI benchmark scenarios
• Help create realistic multi-step business and operational problem-solving tasks

Requirements

• 3–10 years of experience in operations, analytics, consulting, business systems, or related domains
• Strong analytical thinking and operational problem-solving skills
• Experience working with operational workflows, reporting systems, CRM tools, or business analytics
• Good understanding of cross-functional business processes and dependencies
• Experience with spreadsheets, dashboards, operational reporting, or workflow automation
• Strong written communication and documentation skills
• Exposure to AI systems, automation platforms, or evaluation workflows is preferred
• Ability to design realistic and structured operational scenarios for evaluation purposes