Logo for Patronus AI

Domain Expert, AI Evaluation Task Writer: Sales (Contract, Remote)

Role overview

Qualifications

  • 3+ years of hands-on experience in sales or sales operations
  • Strong spreadsheet skills and ability to reconcile data
  • Clear, precise writing in professional English
  • Ability to work independently and asynchronously

Responsibilities

  • Build realistic evidence files from provided scenarios
  • Write task requests and scoring criteria for evaluations
  • Solve tasks end to end and document reference solutions
  • Adjust task difficulty and check for ambiguity in tasks

Key facts

  • Remote from: United States
  • Fixed term
  • Mid-level (2-5 years)
  • English

Hard skills

Other skills

  • Sales
  • Forecasting
  • Negotiation
  • Spreadsheets
  • Time Management

About the company

Patronus AI logo

Patronus AI

  • AI & Machine Learning
  • Startup
  • 11–50
Patronus AI is the leading automated AI evaluation and security company. Our world-class platform enables enterprise development teams to score LLM performance, generate adversarial test cases, benchmark LLMs, and more. Customers use Patronus AI to detect LLM mistakes at scale and deploy AI products safely and confidently. Founded by machine learning experts from Meta AI and Meta Reality Labs, Patronus AI is on a mission to boost enterprise confidence in generative AI. We are backed by Lightspeed Venture Partners, Replit CEO Amjad Masad, Gokul Rajaram, and Fortune 500 executives and board members.

Your match analysis

See how your profile stacks up against this role.

We compared the job requirements to your profile to show where you're strong and where you fall short.

Job description

About Patronus AI

Patronus AI is a frontier lab developing simulation research and infrastructure to accelerate progress toward human-aligned AGI. We are on a mission to simulate all of the world’s intelligence.

We are the team behind some of the earliest and most influential research in AI evaluation like FinanceBench, Lynx, SimpleSafetyTests, CopyrightCatcher, Humanity’s Last Exam, and more. We are formerly AI researchers and engineers from companies like Meta AI, Amazon AGI, and Google. Our customers include foundation model labs and Fortune 500 enterprises like Adobe. We are backed by top-tier investors like Lightspeed Venture Partners, Notable Capital, Stanford University, Noam Brown, Gokul Rajaram, and more.

At a glance

  • Engagement: Independent contractor, fully remote, asynchronous
  • Pay: $50-150/hr for task authoring, paid when the full task is submitted, plus a potential bonus for each accepted task. Strong performers move up within that range or onto productivity-based pay. The trial task is paid.
  • Commitment: 10+ hours per week
  • Start: Rolling, as soon as you pass the trial task
  • Location: Remote

The role

You'll design realistic sales tasks that a qualified professional could solve with the right knowledge and tools, and that today's AI models get wrong. Each task is built around a scenario we provide. You create the working documents someone in your role would actually handle, write the request, solve it yourself and define how the answer gets graded. Your work becomes evaluation and training data for frontier models.

You don't need an AI background. You do need to be the kind of sales professional who notices the deal marked Commit that has no champion and no signature date.

Areas we're hiring for

  • Prospecting and qualification: SDR and BDR work from ICP and account research through lead qualification and routing, outbound sequences and call plans, discovery preparation and the handoff to an account executive
  • Deal execution and closing: account executive work from discovery and qualification (MEDDICC, SPICED or similar) through multi-threaded deal plans, proposals, pricing and discount approvals, order forms, negotiation and close
  • Renewals and account management: post-sale ownership of a book of business, including onboarding handoff, health and usage reviews, renewal quotes and true-ups, expansion and upsell, churn-risk escalation and business reviews
  • Sales operations: the systems and numbers behind the other three areas, including pipeline and forecast rollups, CRM data integrity and stage exit criteria, territory and quota planning, lead routing rules, commission and SPIFF calculation and sales reporting

What you'll do

  • Build realistic evidence files from a provided scenario: CRM opportunity and activity exports, pipeline and forecast reports, lead lists and routing rules, prospect email threads and call notes, proposals and pricing sheets, quotes and order forms, discount approval requests, renewal schedules and usage reports, territory maps, quota plans and commission statements
  • Write clear task requests that state what the deliverable is and how uncertainty and justified assumptions should be handled
  • Solve each task end to end and document a reference solution with the reasoning behind every step
  • Write explicit scoring criteria that let two independent graders reach the same verdict, including acceptable alternative answers and the limits of what the solver is authorized to offer or approve
  • Run tasks against current models, find where and why they fail, and adjust difficulty
  • Check your own and peers' tasks for ambiguity, errors, answers exposed in the materials and shortcuts that skip the intended reasoning
  • Incorporate reviewer feedback within agreed turnaround times and flag guideline gaps or edge cases you run into

Who we're looking for

Required

  • 3+ years of hands-on experience in one or more of the four areas above, as an SDR or BDR, account executive, account manager or customer success manager, or a sales, revenue or deal desk operations analyst or manager. You need to have carried the number or run the process yourself, not only managed people who did.
  • Working command of the methods, rules and systems in your area (for example a qualification framework such as MEDDICC or SPICED, forecast categories and stage exit criteria, discount approval matrices and non-standard terms, commission plan mechanics and territory rules)
  • Strong spreadsheet skills, and the habit of reconciling pipeline, quota and commission figures to their source
  • Clear, precise writing in professional English
  • Ability to work independently and asynchronously against deadlines

Nice to have

  • Experience in B2B SaaS, enterprise or mid-market selling, across more than one motion (inbound, outbound, partner or product-led)
  • Experience with the systems used in your area, such as Salesforce or HubSpot, Outreach or Salesloft, Gong or Chorus, Clari, Salesforce CPQ or DealHub, and CaptivateIQ or Xactly
  • Sales enablement, training or onboarding experience, or having written playbooks, call scorecards or certification exams
  • Prior experience writing or reviewing AI evaluation tasks, exam questions, case studies or training material
  • Familiarity with how AI models fail (invented customer facts, miscalculated discounts or commissions, overconfident forecasts, generic objection handling)

Our standards

  • All tasks, documents and solutions are human-authored. LLM-generated content is not permitted.
  • Tasks and solutions must be original. Nothing may be copied from published or easily searchable sources.
  • Never use real customer, prospect, employer or personal data, or anything covered by an NDA or confidentiality obligation. All companies, people and scenarios are fictional.
  • Quality is judged on accuracy, rubric validity, difficulty and reviewer agreement, not volume.

How hiring works

  1. Short AI interview and resume review
  2. One paid trial task, scored against a fixed quality checklist
  3. Onboarding and your first project

We aim to make a decision within two weeks of your application.

Patronus AI is an equal opportunity employer. We celebrate diversity in our workplace, and all qualified applicants will receive consideration for employment without regard to age, ancestry, color, family or medical care leave, gender identity or expression, genetic information, marital status, medical condition, national origin, physical or mental disability, political affiliation, protected veteran status, race, religion, sex (including pregnancy), sexual orientation, or other legally protected characteristics.

By clicking ‘Apply’, you agree to Greenhouse's Terms of Service and Privacy Policy.

By clicking 'Apply', you agree to Patronus AI, Inc. Privacy Policy.

Related jobs

See all jobs

Other jobs at Patronus AI