AI Evaluation Manager

Location: Nairobi

Position type: Contract role until end of March 2027 (possible extension)

About the Team

GSMA Mobile for Development (M4D) is a dedicated global team within the GSMA, which brings together our mobile operator members, tech innovators, the development community and governments, to realise the impact of mobile in low- and middle-income countries: https://www.gsma.com/mobilefordevelopment/  

The GSMA Innovation Fund accelerates mobile tech solutions that are addressing key global challenges. Through grant funding and tailored technical assistance, we position innovative organisations in low- and middle-income countries (LMICs) to scale operations and amplify social and environmental impact. 

About the Role

This role sits at the intersection of AI and impact measurement. The AI Evaluation Manager will help the Innovation Fund understand how our grantees’ AI solutions work, whether they perform as intended, and how the AI contributes to the impact we fund. Their focus will be threefold: 

• Evaluate grantees’ AI solutions. Work directly with grantees (start-ups and SMEs in LMICs) to understand their AI stack – the models, data, prompts, retrieval and guardrails behind their products – and assess whether the AI performs as intended (model evaluation) and whether the product engages and retains users (product evaluation).
• Connect AI performance to impact. Develop monitoring, evaluation, and learning (MEL) plans and theories of change that trace the pathway from model performance and product engagement through to user and socio-economic outcomes, and design and lead impact assessments for active grantees (this will involve managing external suppliers).
• Lead the Fund’s thought leadership on AI for impact. Look critically at our portfolio data, develop insights into what works and what grantees need, and regularly share our learnings externally through writing, events and panels – going beyond tracking KPIs to help shape the conversation on how AI delivers impact in LMICs.

Each round of the Innovation Fund has a specific thematic focus. However, the AI Evaluation Manager should expect to work across the following themes: start-ups in LMICs, AI and emerging tech, socio-economic impact, and reaching low-income and vulnerable communities. 

Example responsibilities: 

• Reviewing grantees’ AIsolutions to understand their architecture (e.g. foundation models, prompting, fine-tuning, retrieval-augmented generation, agents and guardrails) and how each component contributes to the intended user outcomes.

• Designing and running, or supporting grantees to run, Level 1 model evaluations: building evaluation datasets and reference answers, defining metrics such as accuracy, relevance, safety and hallucination rates, and using LLM-as-judge and human review to test performance.

• Supporting product evaluations: defining and analysing usage, engagement and retention metrics, and helping grantees use A/B testing and product analytics to improve their solutions.

• Advising grantees on practical, proportionate AI evaluation approaches and building their capacity, drawing on the CGD / Agency Fund Generative AI Evaluation Playbook and other emerging standards.

• Drafting MEL plans for each grantee, including AI-specific metrics and a theory of change linking AI performance to outcomes, and supporting MEL delivery throughout the grant cycle.

• Owning and managing grantee KPI reporting, working across the Innovation Fund team to ensure data drives decision-making and support for grantees.

• Developing research plans for grantee baseline and endline surveys and working with suppliers to execute these surveys.

• Developing and building the Innovation Fund’s evidence base on how AI contributes to impact and scale: what is working, for whom, and why.

• Providing strategic leadership on MEL and AI evaluation approaches, helping shape how the Fund measures and understands impact across its portfolio, with reference to existing benchmarks, frameworks and best practices, including emerging AI evaluation frameworks.

• Analysing portfolio-level data to pull out trends, insights and lessons learned.

• Producing regular learning products and thought leadership pieces (reports, blogs, briefs) on AI for impact, drawing on evidence from across the grantee portfolio.

• Representing Mobile for Development externally as a credible voice on AI evaluation and impact, including speaking at events, webinars and panels, and building relationships within the GSMA and with funders, researchers and the wider AI-for-development community.

About You

The ideal candidate combines a solid understanding of the AI stack with a grounding in social impact measurement. You will be comfortable getting under the hood of a grantee’s AI solution, asking the right technical questions, and judging whether it is performing well and contributing to impact. You don’t need to be a machine learning engineer, but you should be able to evaluate an AI model and product with confidence and explain your findings clearly to both technical and non-technical audiences. 

Required skills include:

• Strong working understanding of the AI stack, particularly generative AI and LLM-based applications (e.g. foundation models, prompting, retrieval-augmented generation, fine-tuning, agents and guardrails), and how these are built into products.

• Hands-on experience evaluating AI models and products, e.g. building evaluation datasets, defining performance and safety metrics, using LLM-as-judge or human annotation, and analysing usage and engagement data.

• Familiarity with social impact measurement, e.g. theories of change, MEL frameworks, and survey-based or experimental evaluation methods.

• Experience working with start-ups, SMEs or tech teams, ideally in LMICs across Africa and Asia, in an evaluation, data, product or MEL capacity.

• Ability to think strategically about AI evaluation and impact measurement, and to bring a proactive, thought leadership mindset to understanding and communicating the Innovation Fund’s impact.

• A track record of producing thought leadership (reports, blogs, briefs or learning products) and confidence presenting and speaking publicly about findings to a range of audiences.

• Experience building relationships with key stakeholders internally and externally and communicating findings to a range of audiences. 

• Strong data analysis skills, with the ability to go beyond reporting KPIs to identify trends, draw out insights and translate these into clear, actionable findings. 

• Proficiency in Excel; experience with Python, R or SQL, or with AI evaluation and observability tools, is a strong plus.

• Strong communication skills, especially for interacting with suppliers and grantees (this includes CEOs, CTOs, and directors of companies). 

Desired skills: 

• Experience designing and conducting surveys or other primary data collection, ideally with sample sizes above 100 respondents. 

• Previous experience providing training or capacity building, on any topic, is a plus. 

• While the candidate will need to draft MEL plans for grantees, there is room for training on this activity.  

Contract type

Worker type

What We Offer

Working at the GSMA offers you unparalleled access to the mobile industry. We offer a chance to truly shape the direction of mobile, whatever your role. By joining the GSMA, you will be exposed to a fast-paced rapidly evolving environment, working on global solutions, genuinely fascinating and industry-changing projects and a stimulating and dynamic environment designed to enable you to flourish.

In addition to architect-designed offices and competitive compensation, our benefits include fantastic learning & development opportunities, generous holiday allowances, four additional days off for professional development and many others.

To learn more about the GSMA, visit our career site, our LinkedIn page and our Twitter page.

Being You at the GSMA

We care deeply about diversity, equity and inclusivity and aspire to be the best at it. Your well-being and work/life balance is important, so flexi-time and remote working is available to all staff. We're keen to ensure everyone is equal, represented and connected so we particularly encourage applications from all demographics. The sucess of the GSMA year on year will continue to be contributed by people from all walks of life.

GSMA Values

Our values not only drive our culture – they shape how we work and interact inside and outside our global organisation.

Passionately driven

We approach everything we do with unparalleled capability, tenacity and commitment, knowing that the challenging scale, pace and complexity of our work is what leads to its world-changing impact.

Insightful leaders

We continually develop and engage our expertise, insight and creativity so that we’re always ready to respond to the changing landscape with authority, agility and nuance.

Stronger together

We lean on each other so the industry can lean on us, embracing our diversity by actively seeking out perspectives and skill sets beyond our own, fuelling each other’s successes and constantly asking how we can help.

Underpinning our values is our collective mindset to show up purposefully as good human beings every day, in every situation. When we’re at our best – we are collaborative, considerate and compassionate to others, and we create a safe space for one another to thrive, assuming positive intent in our colleagues. And if we aren’t at our best and the pressure is on – we feel free to be ourselves but still remain curious, lean into the tough stuff and we are always respectful to others and accountable for the part we play.