We are sharing a specialised full-time consulting opportunity for experienced software engineers with strong Python development, debugging, version-control, technical documentation, and AI-assisted coding experience.
This role supports the development of advanced agentic evaluation benchmarks for frontier AI models. Selected professionals will design, implement, and review realistic multi-step software engineering tasks that test the capabilities of AI coding agents across Python development, environment setup, tooling, debugging, and technical problem-solving.
Key Responsibilities
Software Engineering Task Design
- Create realistic, multi-step software engineering challenges based on practical development workflows
- Design technically demanding problems that require implementation, debugging, environment configuration, and analytical reasoning
- Define clear requirements, constraints, expected outputs, and acceptance criteria
- Ensure tasks assess genuine software engineering capability rather than superficial code generation
Reference Solution Development
- Build complete and verifiable reference solutions in Python
- Create the supporting setup, dependencies, tests, and validation checks required for each task
- Write clean, readable, and maintainable code
- Confirm that solutions run reliably within the intended technical environment
- Document implementation decisions and expected behaviour clearly
AI Coding Agent Evaluation
- Use AI coding assistants and agent-based development tools within practical engineering workflows
- Evaluate how frontier models approach complex coding and debugging tasks
- Identify implementation errors, unsupported assumptions, inefficient approaches, and incomplete solutions
- Analyse where AI agents succeed, struggle, or exploit unintended shortcuts
- Document failure patterns and provide evidence supporting evaluation conclusions
Peer Review & Task Refinement
- Review tasks and reference solutions created by other software engineering specialists
- Assess clarity, correctness, difficulty, reproducibility, and technical fairness
- Identify ambiguous instructions, hidden assumptions, grading gaps, and environment issues
- Provide actionable feedback that improves task quality and benchmark reliability
- Collaborate closely with researchers and fellow task authors
Ideal Profile
Strong candidates may have:
- At least 1 year of experience in software engineering, research engineering, or a related coding-intensive role
- Strong hands-on Python scripting, implementation, and debugging skills
- Experience developing clean, readable, and maintainable software
- Everyday fluency with Git, IDEs, repositories, and standard software development workflows
- Comfort configuring environments, dependencies, tooling, and validation processes
- Strong technical writing and documentation skills
- Ability to work independently through ambiguous and open-ended engineering problems
- Reliable availability for approximately 35 hours per week
Educational Background
- An MSc or PhD in computer science, software engineering, another STEM discipline, or a related technical field is highly relevant
- Equivalent practical experience in a research-intensive or engineering-intensive role may also be considered
- Academic or professional work involving significant coding, data analysis, or technical experimentation may strengthen an application
- Open-source contributions, technical projects, publications, or substantial software development work may also be valuable
Nice to Have
- Experience using AI coding assistants, prompt engineering methods, or agent-based workflows
- Previous work in AI training, model evaluation, or benchmark development
- Background authoring technical tasks, reference solutions, or grading criteria
- Familiarity with automated testing, CI/CD workflows, containers, or reproducible environments
- Experience reviewing code or technical assignments created by other engineers
- Knowledge of agentic AI systems and multi-step coding evaluations
- Strong ability to identify edge cases, unintended shortcuts, and subtle implementation issues
- Experience collaborating with AI research or evaluation teams
Why This Opportunity
- Apply practical software engineering expertise to frontier AI evaluation
- Design realistic coding tasks grounded in professional development workflows
- Help researchers understand where advanced AI coding agents succeed and fail
- Work across Python implementation, debugging, environment setup, and benchmark development
- Collaborate closely with AI researchers and experienced software engineers
- Participate in a structured full-time remote role with competitive hourly compensation
Contract Details
- Full-time W-2 contingent employment opportunity
- Fully remote within the United States
- Expected commitment of approximately 35 hours per week
- Competitive rates between $55–$85 per hour depending on expertise and project scope
- Individual tasks may require one to two days of focused engineering work
- Work may include task design, Python development, reference-solution creation, AI agent evaluation, peer review, and technical documentation
- Engagement scope and duration may evolve according to project requirements and performance
About the Platform
This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.
By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy.
