Logo for Epoch AI

Researcher, Evaluations

Epoch AI

Role overview

Qualifications

  • Analytical thinking
  • Grounded, skeptical mentality
  • Comfort with AI agents and tools
  • Familiarity with AI benchmarks and evaluations

Responsibilities

  • Create and curate an evaluation suite
  • Evaluate AI systems
  • Communicate your research
  • Conduct data analysis

Key facts

Hard skills

AI Testing Computational Tools Python (Programming Language)

Other skills

  • Analytical Thinking
  • Verbal Communication Skills

About the company

Epoch AI logo

Epoch AI

Research & Scientific Services

Epoch AI is a multidisciplinary research institute investigating the trajectory of Artificial Intelligence (AI). We scrutinize the driving forces behind AI and forecast its ramifications on the economy and society. We emphasize making our research accessible through our reports, models and visualizations to help ground the discussion of AI on a solid empirical footing. Our goal is to create a healthy scientific environment, where claims about AI are discussed with the rigor they merit.

Company details

IndustryResearch & Scientific Services
Company size11 - 50

Links

Website LinkedIn See all jobs

Your match analysis

See how your profile stacks up against this role.

We compared the job requirements to your profile to show where you're strong and where you fall short.

Job description

Epoch AI is looking for a researcher to evaluate frontier AI models on hard-to-grade tasks drawn from real-world scenarios.

About the role

We’re seeking a Researcher to lead a new effort evaluating how well frontier models perform on the kinds of open-ended tasks that make up real office work. You will curate a suite of realistic tasks to serve as a benchmark, design the grading rubrics for AI performance, and run newly-released models through the suite, assessing their performance both quantitatively and qualitatively.

The focus is on how models handle messy, real-world work rather than on scientific knowledge or programming ability. The role makes heavy use of AI tools, but strong software engineering experience is not required. Comfort setting up AI-assisted automated workflows is a plus.

If this role sounds interesting, we are also looking for researchers on multiple other teams. 

Applications are rolling. 


Key Responsibilities
  • Create and curate an evaluation suite. Find real-world tasks that serve as challenging tests for practical AI capabilities, and update the tasks over time as AI capabilities evolve. Devise rubrics for evaluating AI performance.
  • Evaluate AI systems. Regularly evaluate new, notable AI models and products on the task suite. Update tasks and rubrics to reflect the changing landscape of AI capabilities.
  • Communicate your research. Create public-facing reports, blog posts, and data visualizations with your observations. Ensure the evaluations feed into our other research topics and help keep our team informed. 
  • Conduct data analysis. Analyze evaluation results and compare models across tasks. 
  • Improve the process. You might automate parts of the workflow, and build out parts of the evaluation into standalone benchmarks. 

    • What we are looking for
  • Analytical thinking. You conduct experiments with rigor and care, making sure that findings are well-supported by evidence.
  • Grounded, skeptical mentality. You form your own well-reasoned view of what an AI system can do, distinguishing practical capabilities from hype.
  • Comfort with AI agents and tools. You have experience working with AI agents in the course of your own work, and are comfortable delegating tasks.
  • Familiarity with AI benchmarks and evaluations. You follow AI capabilities at least casually and have opinions on what benchmarks do and don’t tell us.
  • Research and data-analysis experience, including enough comfort with light coding to analyze your own results.
  • Strong written communication skills: You can convey nuanced observations clearly and precisely.

    • Nice to have
  • Experience testing frontier models and writing assessments of their capabilities
  • Coding skills, including python proficiency
    • If you don’t tick all these boxes but think you would be a great fit, please consider applying anyway!

    Compensation & Benefits
  • Annual salary between $115,000 – $200,000 USD, depending on location and experience. 
  • Salaries are not restricted to USD, and contracts and payments are usually in local currencies. Conversions are based on one-year average exchange rates.
  • Fully remote environment, including flexible work hours. 
  • Competitive global benefits program, including a comprehensive health insurance program—including supplemental benefits specific to a local country, as available and mandated by local law—and life insurance and a pension plan, if applicable in your country.
  • Generous paid time off (PTO), including no specific annual limit, with 30 days PTO per year protected, unlimited personal and sick leave, and 4 months paid parental leave for permanent staff with at least 12 months of tenure (prorated parental leave if less than 12 months). 
  • A flexible and generous expense policy for you to spend on equipment and a large range of productivity tools or learning/development opportunities, including unlimited spending on AI tools, subject to regulations and manager approval. 
  • Paid work trips, including 3 staff retreats per year and relevant conferences.
  • Access to our very well-equipped offices in Berkeley, California, including paid meals, snacks, gym, and more. All staff, independently of where they are based, have access to the office for at least 20 days each year.

    • Additional Information

    While we welcome applicants from all time zones, we prefer candidates who can overlap with UTC–8 (Pacific Time) and UTC (Greenwich Mean Time), as most of our staff work in this range of time zones. We also prefer candidates who can travel: we hold three retreats per year, during which we record podcast episodes and other communication efforts.

    Please submit all of your application materials in English and note that we require professional level English proficiency.

    Epoch is committed to building an inclusive, equitable, and supportive community for you to thrive and do your best work. We’re committed to finding the best people for our team, so please don’t hesitate to apply for a role regardless of your age, gender identity/expression, political identity, personal preferences, physical abilities, veteran status, neurodiversity or any other background. Please email careers@epoch.ai if you have any questions about this role, accessibility requests, or if you want to request an extension to the application deadline. However, we will not review applications submitted to this email address; please submit your application through the link on this page. 

    About Epoch AI

    Epoch AI is a research institute that investigates trends in machine learning and the economic consequences of AI. Our mission is to develop a comprehensive, publicly accessible knowledge base on AI that informs policymakers, industry leaders, and society at large.

    We strive to achieve both rigor and accessibility to our work, as exemplified by some of our most successful projects, including our database of AI models and our AI trends dashboard. Our body of research includes our work on compute trends (IJCN 2022), data scarcity (ICML 2024), and algorithmic progress (NeurIPS 2024). You can read more about our work and mission on our website and in this Time profile.

    Apply once. Then go straight to the hiring manager.

    After you apply, unlock the direct contact details of the people who actually make the call. A quick follow-up makes you 5x more likely to land an interview.

    MR

    Marcus Rivera

    Chief Revenue Officer

    m.rivera@company.com
    linkedin.com/in/marcusrivera
    Unlocked after you apply
    ·

    Researcher Related jobs

    United StatesResearcher

    Other jobs at Epoch AI

    Premium

    Reach out to the hiring manager directly.

    Gain access to the contact details of the hiring managers who actually decide, and reach out to network with them directly. That, plus more when you upgrade:

    • Full match report with fit score and gaps
    • Career diagnostics on how recruiters read you
    • Curated company matches and warm intros
    • 48h early access to new roles

    Cancel anytime.