We are sharing a specialised part-time opportunity for licensed physicians in active clinical practice to contribute to advanced medical AI evaluation through clinical reasoning, scenario design, benchmark development, and structured assessment of model performance.
Selected physicians will work with research teams to evaluate how AI systems approach realistic medical problems, identify gaps in clinical reasoning, and help develop assessment methods that reflect the complexity and nuance of evidence-based practice. The opportunity is open across medical specialties.
Key Responsibilities
Clinical Reasoning Evaluation
- Evaluate AI performance on realistic medical reasoning tasks
- Assess whether clinical conclusions are appropriately supported
- Identify gaps in medical knowledge or decision-making
- Review model reasoning against evidence-based practice
- Distinguish clinically significant errors from minor limitations
Clinical Scenario Development
- Create realistic medical scenarios for model evaluation
- Design cases that require nuanced clinical judgement
- Incorporate diagnostic and management decision points
- Develop scenarios reflecting real-world clinical complexity
- Ensure cases meaningfully test model capabilities
Evaluation Framework Design
- Design systematic assessment frameworks for medical AI systems
- Define criteria for evaluating clinical reasoning quality
- Develop methods that capture nuance in medical decision-making
- Support consistent evaluation across repeated clinical tasks
- Refine assessment approaches based on observed model performance
Medical Knowledge & Decision Analysis
- Identify areas where models demonstrate weak clinical understanding
- Evaluate diagnostic reasoning and treatment-related judgement
- Assess model responses for clinical relevance and completeness
- Apply specialty-specific expertise where appropriate
- Provide structured rationale for evaluation decisions
Research Collaboration & Feedback
- Collaborate with research teams on medical AI evaluation
- Communicate clinical findings clearly and concisely
- Provide actionable feedback on model strengths and limitations
- Contribute to strategies for improving medical model performance
- Support development of clinically meaningful benchmarks
Ideal Profile
- Licensed physician currently engaged in active clinical practice
- MD, DO, or equivalent medical qualification
- Open to physicians across medical specialties
- Relevant backgrounds may include Public Health, Pediatrics, Emergency Medicine, Obstetrics & Gynecology, or Gynecological Oncology
- Strong clinical decision-making experience
- Solid understanding of evidence-based medicine
- Ability to assess complex medical reasoning objectively
- Strong analytical and critical-thinking capabilities
- Excellent written and verbal communication skills
- Comfortable explaining nuanced clinical decisions clearly
- Able to identify gaps in medical knowledge or reasoning
- Interest in the application of AI to clinical practice
- Comfortable collaborating with research teams in a remote environment
- Able to work independently around existing clinical commitments
Engagement Details
- Part-time remote engagement
- Flexible scheduling
- Commitment of up to 30 hours per week
- Initial project duration of approximately 1 month
- Extension may be available depending on performance and project fit
- Work will focus on clinical reasoning evaluation, scenario development, and medical benchmark design
- Physicians from any clinical specialty may be considered
- Scheduling is designed to accommodate ongoing clinical commitments
- Compensation is not specified in the source materials
- Work must be completed without using confidential, proprietary, patient-identifiable, protected health, unpublished clinical, or otherwise restricted information belonging to any patient, employer, healthcare organisation, research institution, client, or other third party
About the Platform
This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.
By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy
