Meta Analysis Specialist

About Us:

DevTech Systems, Inc. (DevTech) is an international consulting firm dedicated to development, with 40 years of experience providing advisory services and technical assistance to government, private sector, and civil society stakeholders in more than 100 countries. DevTech core practice areas include: Monitoring Evaluation, Research and Learning; Data Solutions; Public Financial Management and Fiscal Sustainability; and Education, Gender, and Youth.

 

At DevTech, we care deeply about doing work that leads to positive change in the world. We value diverse perspectives and are committed to an inclusive work environment. We encourage each person to learn, develop, and meet their professional potential. We rethink and evolve how we do things to grow and improve our company.

Overview:

Under the Education Technical Support Services contract (ETSS), DevTech is seeking a Meta Analysis Technical Specialist with experience using Cochrane or Campbell guidelines to advise on the design, implementation, analysis, and reporting of findings of two systematic reviews of questions #2 and #12 of the USAID Latin America and the Caribbean Education Learning Agenda.

 

Q2. How education influences irregular migration. Build on existing systematic review ensuring all databases or gray literature is synthesized.

 

Q12. What workforce development models after secondary education or higher result in youth employability. Full systematic review required.

 

USAID would like to expand on the rapid-evidence stock-taking that the University of Notre Dame did to conduct a full systematic review and meta-analysis of findings related to each of the learning agenda questions, following the guidance from the Cochrane Collaboration on rigorous systematic reviews. This SOW is focused on Q2 and Q12 as the most pressing to get answers to through consultations with USAID’s LAC Missions and the LAC Education Learning Hub.

 

The USAID LAC Education Team, USAID Missions, USAID IPI/EDU, LAC Ministries of Education, USAID implementing partners, and other donors will use the systematic review findings to inform the design and implementation of future education interventions in the LAC region. These actors, and especially USAID, will also use the study to inform decisions about priority research questions, topics, contexts moving forward to round out the evidence base. As such, the systematic review should include both findings related to the questions as well as details about what major gaps exist in the research with regards to findings, methods, contexts/locations, etc.

 

Responsibilities:

The Meta- Analysis Senior Technical Specialist (TS) will provide guidance on the MA design and methods. The TS shows expertise in meta-analysis and implementation of Cochrane Collaboration guidelines or similar. The TS will oversee the methodological approach and meet with the team after every research milestone completion to ensure that the team follows all guidelines. The TS will provide the shell tables for the Results section of the report and will review the do files for statistical analysis as well as provide an outline for the Discussion section of the document and the report recommendations.  The TA reports to DevTech’s Education, Gender, and Youth Deputy Director/LAC ETSS Technical Advisor.

 

Scope of Work:

USAID’s Bureau for Latin America and the Caribbean (LAC Bureau) commissioned the LAC Education Learning Agenda through the USAID-University of Notre Dame (UND) Supporting Holistic and Actionable Research in Education (SHARE) Cooperative Agreement, with the goal of forming questions to help the USAID LAC Bureau, other donors, and implementing partners better understand the best way to support both education recovery from COVID-19 and inclusive education for marginalized populations such as those from 8 lower socioeconomic classes, migrants, Indigenous children and youth, learners with disabilities, those from rural areas, and those from other minority groups. The timing for the development of the learning agenda also coincided with a strong U.S. government focus on addressing the root causes of irregular migration and violence in LAC through early childhood education/ development and youth and workforce development.

 

To create the learning agenda, UND engaged with a key working group of USAID staff from Washington and USAID’s LAC Missions to define the parameters of the agenda and the process. The process included wide consultations with the LAC Education Learning Hub; key informants from implementing partners, Ministries of Education, and USAID.

 

As of 2023, the LAC countries and subregions with USAID education programming (bilateral and/or regional) are the Dominican Republic, 11 countries in the Eastern and Southern Caribbean, Ecuador, El Salvador, Guatemala, Haiti, Honduras, Colombia, and Paraguay. The Learning Agenda will contribute to programming in these countries and inform the priorities of future activities and interventions.

 

The purpose of the USAID LAC Education Learning Agenda is to define institutional learning priorities across the region, linked to programming, decision-making and collaboration efforts. The learning agenda will help USAID to prioritize its primary research, secondary research syntheses, and evidence dissemination efforts until at least 2028. USAID also plans to use the evidence gathered and synthesized from the agenda to guide programming and hopes other donors and implementing partners will also do so.

 

The final learning agenda has 12 questions and is available here.  In addition to developing the learning agenda, the UND team also conducted a quick and high-level rapid evidence stock-taking to determine the level of existing evidence already available to answer each of the 12 learning agenda questions. They used this stock-taking to classify according to the following framework:

 

 

Generation: Minimal evidence exists for these questions. The generation of new evidence is required.

Capture: Evidence exists for these questions but is dispersed; Capture and analysis of new evidence are needed.

Dissemination: Evidence exists in a usable form, but it is relatively unkown and requires targeted distribution.

 

They found that none of the questions were at the point of dissemination.  They also used the stock-taking to provide a high-level overview of findings from existing evidence for each question (see document for high-level details and the rapid evidence stock-taking document for a list of studies referenced and their key findings).

 

1. SYSTEMATIC REVIEW METHODS:

USAID seeks two separate rigorous systematic reviews and meta-analyses (to the extent that sufficient studies exist with similar outcomes to inform such meta-analyses; please note that more than one meta-analysis may be needed per systematic review, depending on the number of outcomes identified; DevTech should prioritize the top three or fewer from each systematic review) – one to answer each of the learning agenda questions. The reviews should follow the guidance from the Cochrane or Campbell Collaborative, as linked above. For the migration question, please note that this systematic review is already advanced and just needs a few researchers to check if there are any untapped databases or gray literature that can be included and then to synthesize the results of the study.  Please find, here, a set a documents to help get this review finalized:

• LAC Learning Agenda Implementation Plan – Please see the tab on Existing Studies – where you will find a list of already-pulled studies for all learning agenda questions, including more than 100 studies related to learning agenda question 2. Note, that each of these studies also has details about methods, findings, limitations, etc. These studies should be used to synthesize findings and to determine if any meta-analyses are possible.
• LAC Education and Migration Inclusion Criteria – Please see this document for details on the systematic review process and inclusion/exclusion criteria. DevTech can use this document to determine if there are any untapped resources, including gray literature, that should be consulted to ensure a full and complete body of evidence.
• A short, high-level, summary of the evidence and findings that the researchers who did the data collection for this study think should be included in study.
• The rapid evidence stock-taking document. Please also make sure that all of the studies from this review are included in both reviews.

For the second question, on workforce development, USAID is requesting a full systematic review from start to finish, noting that some documents are available in the implementation plan and rapid evidence stock-taking documents referenced above. All studies included should be checked against a rigorous set of quality criteria to ensure their inclusion is warranted.

The task order is divided into two phases.

Phase 1 will focus on conducting a systematic review of the literature, assessing the risk of bias, and determining if a full meta analysis is possible. If Phase 2 is possible, it would focus on completing the meta analysis and developing a full report for each RQ as follows:

	Activity		Deliverable
Phase 1	1	Review background documents. Gain familiarity with USAID Learning Agenda and available information on RQ 2 and 12 on Migration and Workforce Development. Review of the preliminary report that Notre Dame developed on RQ2- Migration.	n/a
	2	Approve Study Protocol. Review study methodology and provide guidelines on the use of Cochrane checklist. PICOS - population, intervention, contrafactual, observation, study design.	Study Design/protocols for RQ2 Study Design/protocols for RQ12
	3	Conduct post screening of titles and abstracts review, check in, y/n study. For each title that you retain, you move to the second screeining with the full text. At that stage you will see if your inclusion/exclusion criteria is met or not. Studies that you keep are the ones that become part of the risk of bias group . [regional advisor will scan the list and will use a random sample to verify]	List of titles and abstracts RQ2 List of titles and abstracts RQ12
	4	Risk of bias scoring and validation. Assess if the quality of evidence will allow for any confidence statement about the size of effect using experiments and quasi-experiments. Make high, low, unclear dimension on the bias assessment.	Individual studies risk of bias and aggregate risk of bias report. RQ2  Individual studies risk of bias and aggregate risk of bias report. RQ12
	5	Recommendations for or against Meta Analysis. If for, and revised Study Protocol with eligibility criteria, specific outcomes to analyze, plan to address issues of bias, and minimum standards required for the meta-analysis.	Two preliminary reports with final list of studies that qualify based on the bias rating, risk of bias tool explanation and assessment, and recommendations to either move forward or not.
Phase 2	6	Data extraction and effect size validation. Review effect size definitions, if ES not available, determine if proxies can be created with available data.	Codes for effect size validation
	7	Conduct Cochrane rating method-  GRADE approach (Grading of Recommendations, Assessment, Development, and Evaluation) for assessing certainty (or quality) of a body of evidence. Effect size, variability and quality of evidence.	GRADE review
	8	Approve final report’s Table of Content (TOC) with heathers and sub heathers	TOC
	9	Review report draft and validate findings
	10	Approve final report	Final Report

Qualifications:

• Master’s degree or higher in Economics, Education Policy, Program Evaluation, or related field.
• Experience leading at least one past systematic review and meta-analysis following Cochrane’s Collaboration or other similar guidance- a must.
• Quantitative research experience assessing large scale data in international development/or education interventions.
• Demonstrated experience in designing and conducting impact evaluations.
• Experience producing tree plots and other representations that seek to report standardized effect sizes across countries.
• Ability to oversee the production of a high-quality assessment report that includes data visualizations.
• Excellent interpersonal skills; and

• Fluency in written and spoken Spanish and English.

 

Preferred Qualifications:

• Practitioner or academic with familiarity with migration and or youth workforce development and education outcomes in the LAC region.

Deliverables, Level of Effort, Schedule:

The Technical Specialist will guide the team in the following activities and deliverables distributing its level of effort and time as follows:

	Activity		Deliverable	Estimated Days	Due Date
Phase 1	1	Review background documents. Gain familiarity with USAID Learning Agenda and available information on RQ 2 and 12 on Migration and Workforce Development. Review of the preliminary report that Notre Dame developed on RQ2- Migration.	n/a	2
	2	Approve Study Protocol. Review study methodology and provide guidelines on the use of Cochrane checklist. PICOS - population, intervention, contrafactual, observation, study design.	Study Design/protocols for RQ2 Study Design/protocols for RQ12	4	9/15/2024
	3	Conduct post screening of titles and abstracts review, check in, y/n study. For each title that you retain, you move to the second screeining with the full text. At that stage you will see if your inclusion/exclusion criteria is met or not. Studies that you keep are the ones that become part of the risk of bias group . [regional advisor will scan the list and will use a random sample to verify]	List of titles and abstracts RQ2 List of titles and abstracts RQ12	3	12/15/2024
	4	Risk of bias scoring and validation. Assess if the quality of evidence will allow for any confidence statement about the size of effect using experiments and quasi-experiments. Make high, low, unclear dimension on the bias assessment.	Individual studies risk of bias and aggregate risk of bias report. RQ2  Individual studies risk of bias and aggregate risk of bias report. RQ12	7	2/15/2025
	5	Recommendations for or against Meta Analysis. If for, and revised Study Protocol with eligibility criteria, specific outcomes to analyze, plan to address issues of bias, and minimum standards required for the meta-analysis.	Two preliminary reports with final list of studies that qualify based on the bias rating, risk of bias tool explanation and assessment, and recommendations to either move forward or not.	5	4/15/2025

Inputs

• LAC Learning Agenda Implementation Plan
• The final learning agenda here.
• LAC Education and Migration Inclusion Criteria
• A short, high-level, summary of the evidence .
• The rapid evidence stock-taking document.

 

EEO Statement:

NOTE: This job posting should not be construed to imply that the requirements are the exclusive standards of the position nor will it be the sole basis for any subsequent employee evaluations. Incumbents will follow any other instructions and perform any other related duties as may be required by their supervisor. 
 
All qualified applicants will receive consideration for employment and will not be discriminated against on the basis of race, color, national origin, religion, sex (including pregnancy, childbirth, or related medical conditions), age (40 and older), physical and mental disability, genetic information (including family medical history) or characteristics, military status, protected veteran status, sexual orientation, gender identity, citizenship or alienage, or any other protected status as established by applicable federal, state, or local law.