AI Evaluator

About the Job:

As an AI Evaluator (both English and Hindi) you will work to help state-of-the-art generative AI models perform better using your language and critical thinking skills. You will be required to ensure quality in a high-investment area for our client’s program to evaluate LLM (Large Language Model) responses to human queries while checking for accuracy and policy sustainability. 

There will be badge tests and an overall training process aimed at verifying the target skills and ensuring that any new candidate has the right aptitude for this work. All activities need to be completed within the defined SLA (service level agreement) and quality. All work will be in both Hindi and English.

This is a fully remote position, and candidates are required to have their own computer with a reliable high-speed internet connection to perform this role effectively.

Key Responsibilities: 

• Proficiency in English and Hindi: intermediate or advanced language skills, which the candidate will use to:
• Analyze and understand the meaning of the model response, the user feedback and how it relates to the response.
• Identify messages that are inappropriate and trigger policy concerns. 
• Define the target information (claims) in the response.
• Recognize information that isn’t clearly pointed out as wrong - e.g. missing details that make the response incomplete - or information that’s mixed up between multiple entities (disambiguation). 
• Formulate appropriate strings for fact-checking and web research that helps them find the right sources quickly.
• Recognize any differences between the text in the input passage and the text generated in the response, thus identifying input issues.
• Understand the context of the user-model interaction, and recognize when the model doesn’t interpret that context correctly.
• Workflow-related: understand complex Guidelines instructions, ask well-formulated questions, leave informative justifications and task comments.
• Understand the client platform and follow team level protocols (like how/when to access the task, etc.).
• Strong logic, intuitive problem-solving and critical thinking; adaptability to the needs of the task. The candidate needs to be open-minded and welcome different viewpoints or solutions.
• Proactiveness, confidence to work independently without constant support. Reliability and being able to make good decisions on their own.
• Be a thought partner to delivery managers, input into the project delivery strategies, and share quality best practices to aid in achieving the project goals

Minimum Qualifications:

• Have completed a Bachelor's degree.
• Fluency in Hindi at a level equivalent to C2 level.
• Fluency in English at C1 level. 
• Digital skills for high levels of research to find accurate information from reliable internet sources.
• Strong critical thinking, reasoning, and exceptional problem-solving skills.
• Excellent attention to detail.
• Experience in GenAI is an added advantage.
• Knowledge of Indian cultural background.

Preferred Qualifications:

• Excellent written and oral communication skills in both English and Hindi.
• Hindi qualifications are preferred at Hindi Honors or MPhil levels.
• Demonstrable ability to perform well in a rapidly changing and extremely global team.
• Passion for our mission of ensuring a world-class support experience for our community and customers.
• Experience as an editor is an added advantage.