We are sharing a specialised full-time opportunity for experienced technical professionals to operate at the intersection of AI research, machine-learning data systems, evaluation, and real-world model performance.
Selected professionals will take ownership of research and evaluation initiatives designed to generate high-quality, defensible research signal and translate that signal into measurable improvements in AI systems. The role combines evaluation design, ML-oriented data development, failure analysis, quality calibration, and close collaboration with researchers, domain experts, and operational teams.
Key Responsibilities
Research Evaluation & Signal Quality
-
Own research and evaluation initiatives from problem framing through data design, quality calibration, and signal validation
-
Define rigorous approaches for determining whether experimental results provide reliable and defensible research signal
-
Analyse model and system failures to identify root causes, edge cases, and opportunities for improvement
-
Evaluate whether datasets, experiments, and conclusions meet appropriate quality thresholds
-
Act as a quality gate when signal strength, data integrity, or supporting evidence is insufficient
ML-Oriented Data & Evaluation Design
-
Design ML-oriented data systems including task definitions, annotation schemas, rubrics, incentives, and supporting pipelines
-
Structure data and evaluation workflows around downstream model-performance objectives
-
Translate ambiguous real-world behaviour into measurable evaluation frameworks and new data categories
-
Identify gaps in evaluation or dataset coverage and recommend where additional investment or iteration is needed
-
Develop quality-assurance processes that maintain strong and consistent research standards
Failure Analysis & Iterative Model Improvement
-
Investigate model and system behaviour to identify recurring weaknesses and performance limitations
-
Iterate rapidly on evaluations, datasets, feedback loops, and quality standards
-
Use experimental findings to guide improvements in model or agent performance
-
Determine when research directions should be expanded, revised, paused, or discontinued based on evidence
-
Maintain a systems-level perspective focused on end-to-end AI performance rather than isolated components
Research Collaboration & Technical Communication
-
Work closely with researchers, domain experts, operators, and cross-functional teams throughout project kickoff, calibration, and iteration
-
Communicate research findings, trade-offs, limitations, and signal strength clearly to technical and non-technical stakeholders
-
Translate research progress into credible narratives grounded in evidence
-
Support alignment between experimental work and real-world system requirements
-
Contribute strong technical judgement in ambiguous, high-impact research environments
Ideal Profile
-
Strong professional judgement regarding research signal quality and whether findings are ready to support broader conclusions
-
Experience designing ML-oriented datasets, evaluation frameworks, annotation systems, rubrics, or QA processes
-
Ability to translate complex and ambiguous real-world system behaviour into structured research and evaluation opportunities
-
Strong ownership mindset and comfort making decisions in uncertain or rapidly evolving environments
-
Excellent written and verbal communication skills
-
Ability to explain technical trade-offs, limitations, evidence quality, and research findings clearly
-
Proven experience working directly with researchers, technical experts, or domain specialists during project calibration and iteration
-
Systems-level understanding of model, agent, or AI-system performance
-
Experience with reinforcement-learning environments, simulators, or feedback-driven training systems is advantageous
-
Experience improving agentic systems or AI systems operating within real-world workflows is beneficial
-
Prior work within applied research or production environments with direct impact on deployed systems is advantageous
-
Experience designing evaluations for complex or real-world tasks is strongly valued
-
Familiarity with expert incentive design or high-stakes technical research programmes is beneficial
Engagement Details
-
Full-time engagement
-
Fully remote
-
Compensation: $600,000–$2,000,000/year
-
Work will span research evaluation, ML-oriented data design, failure analysis, quality calibration, and iterative AI-system improvement
-
Responsibilities may include acting as a quality gate for research claims, datasets, and evaluation results
-
Collaboration will involve researchers, domain experts, operational teams, and other technical stakeholders
-
Project priorities, evaluation frameworks, and research directions may evolve based on experimental findings and system performance
-
Work must be completed without using confidential or proprietary information belonging to any employer, client, institution, or other third party
About the Platform
This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.
By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy
