Remote | Open Source Software Engineer — $60–$80/hour

We are sharing a specialised part-time consulting opportunity for experienced Software Engineers with hands-on open-source contribution or maintainer experience and strong expertise in repository-level code review, testing, debugging, and software quality evaluation.

This role focuses on reviewing software-engineering benchmark tasks for correctness, reproducibility, and grading integrity. Selected engineers will audit repository-level assignments, reference patches, test harnesses, containerised environments, and evaluation logic while identifying technical flaws, unintended shortcuts, and weaknesses in task design.

Key Responsibilities

Repository-Level Code Review

• Review software engineering tasks built around real code repositories
• Assess whether task requirements are technically clear, complete, and reproducible
• Evaluate repository state, dependencies, configuration, and expected behaviour
• Identify ambiguities or implementation issues that could affect task validity
• Apply practical engineering judgement to realistic codebase-level problems

Reference Patch Auditing

• Review reference patches for correctness and completeness
• Determine whether proposed solutions appropriately address the underlying software issue
• Identify unintended behavioural changes, incomplete fixes, or unsupported assumptions
• Compare reference implementations against task requirements and expected outcomes
• Assess whether alternative valid implementations are treated fairly

Test Harness & Grading Review

• Audit test runners and automated evaluation logic
• Assess whether tests accurately measure the intended behaviour
• Identify missing coverage, brittle assertions, or grading inconsistencies
• Verify that evaluation criteria appropriately distinguish correct from incorrect solutions
• Review benchmark tasks for reliable and repeatable scoring

Reproducibility & Environment Validation

• Evaluate whether tasks can be reproduced consistently across clean environments
• Review dependency installation, build processes, configuration, and runtime requirements
• Assess Docker-based isolation and containerised execution
• Identify environmental dependencies or hidden assumptions affecting reproducibility
• Verify that tasks execute reliably under their intended setup

Benchmark Integrity

• Identify potential answer leakage, unintended shortcuts, or reward-hacking opportunities
• Evaluate whether benchmark structure exposes information that makes tasks artificially easy
• Review task and grading design for loopholes or exploitable behaviours
• Assess whether successful completion genuinely demonstrates the intended engineering capability
• Recommend improvements where benchmark integrity is compromised

Software Testing & Debugging

• Investigate failing or inconsistent benchmark tasks
• Review stack traces, logs, test failures, and repository behaviour
• Identify root causes of technical issues
• Distinguish task defects from legitimate implementation failures
• Assess whether debugging and validation processes follow sound engineering practices

Open-Source Engineering

• Apply experience from contributing to or maintaining open-source software
• Evaluate repository conventions, contribution patterns, and realistic development workflows
• Review patches with the perspective of an experienced contributor or maintainer
• Assess whether proposed changes would meet reasonable code-review expectations
• Apply practical judgement derived from real-world pull request and repository experience

Multi-Language Code Evaluation

• Review software written in Python
• Evaluate tasks involving at least one additional ecosystem such as Java, Go, TypeScript, or C++
• Assess code structure, tests, implementation choices, and repository conventions across languages
• Identify language-specific implementation or testing issues
• Apply consistent engineering standards across different technology stacks

Rubric-Based Evaluation

• Assess benchmark tasks against structured technical criteria
• Provide clear written explanations supporting evaluation decisions
• Reference specific code, tests, patches, or execution behaviour when identifying issues
• Apply grading standards consistently across assignments
• Distinguish substantive benchmark defects from minor implementation differences

Ideal Profile

• 3+ years of professional software engineering experience
• Demonstrated open-source contribution or maintainer experience, such as merged pull requests, committer responsibilities, or maintainer roles
• Strong ability to review repository-level software changes
• Experience auditing reference patches, test runners, and automated test suites
• Comfortable evaluating Docker-based isolation and reproducible development environments
• Strong understanding of software testing, debugging, and code-review practices
• Ability to identify answer leakage, reward hacking, or other benchmark-integrity issues
• Strong proficiency in Python
• Professional fluency in at least one additional language such as Java, Go, TypeScript, or C++
• Familiarity with SWE-Bench Verified or similar repository-level software engineering benchmarks is preferred
• Maintainer or contributor history with established Python open-source projects is highly valued
• Previous code-review, software evaluation, or task-grading experience is advantageous
• Strong written communication and ability to provide precise technical feedback

Engagement Details

• Part-time independent contractor engagement
• Fully remote within the United States
• Flexible scheduling based on project requirements
• Compensation: $60–$80/hour
• Work focuses on repository-level software evaluation, reference-patch review, testing, reproducibility, benchmark integrity, and technical quality assessment
• Projects may be extended, shortened, or concluded based on project needs and performance
• Work must be completed without using confidential or proprietary information belonging to any employer, client, institution, or other third party
• H1-B and STEM OPT support is unavailable for this engagement

About the Platform

This opportunity is available through 24-MAG LLC. We connect experienced professionals with remote consulting opportunities across technical, evaluation, and project-based workstreams.

By submitting this application, you acknowledge that your information may be processed by 24-MAG LLC for recruitment and opportunity matching in accordance with our Privacy Policy: https://www.24-mag.com/privacy-policy.