Director, Free Response Rating Capability Analyst

Director, FR Rating Capability Analyst 

College Board – AP & Instruction 
Location: This is a fully remote role. Candidates who live near CB offices have the option of being fully remote or hybrid (Tuesday and Wednesday in office). 
Type: This is a full-time position 

About the Team 

The AP Reading and Scoring team is responsible for delivering high-quality, reliable, and scalable scoring operations for millions of Advanced Placement (AP) free response exam submissions annually. The team partners across Scoring, Reader Management, Psychometrics, Assessment Production, and Technology to continuously improve scoring quality, operational effectiveness, and digital scoring capabilities that support AP’s mission and growth. 

About the Opportunity 

As the Director, FR Rating Capability Analyst, you will play a critical role in designing, building, and improving the operational and analytic capabilities that enable high-quality AP Reading scoring. This role sits at the intersection of scoring operations, data analysis, digital scoring systems, and process improvement. 

You will translate scoring performance insights into scalable tools, workflows, reporting frameworks, and operational improvements that strengthen scoring accuracy, efficiency, and reader effectiveness. You will collaborate closely with cross-functional partners to support continuous improvement across the AP Reading lifecycle and help scale digital scoring capabilities for the future. 

In this role, you will: 

FR Rating Capability Design & Ownership (35%) 

• Define and maintain the capability roadmap for FR rating operations, including tools, workflows, reporting structures, and performance measurement systems that support high-quality scoring at scale.  

• Translate operational needs and scoring insights into clearly defined business requirements, system enhancements, reporting frameworks, and process improvements.  

• Partner with Technology and Product teams to design, test, and implement enhancements to scoring platforms and data pipelines.  

• Develop scalable frameworks for evaluating scoring effectiveness, including consistency, accuracy, throughput, and reader-level performance metrics.  

• Establish and maintain expertise in AP exam item types, rubric structures, allowable response formats, and annual assessment changes that impact scoring operations.  

• Serve as a liaison across Scoring, Reader Management, Assessment Production, Psychometrics, and Technology teams regarding FR rating operational systems and processes.  

Operational Integration & Reading Support (30%) 

• Support the end-to-end AP Exam scoring workflow, including reader training, scoring operations, psychometric data collection, and operational reporting.  

• Partner with cross-functional teams to ensure scoring systems are configured appropriately for annual AP Reading operations across multiple subjects, forms, and item types.  

• Monitor scoring performance signals during live AP Reading and enable rapid operational interventions through data-driven insights.  

• Collaborate with Reader Management and Workforce Operations teams to align reader performance insights with recruitment, training, and retention strategies.  

• Support onboarding and training for stakeholders on scoring tools, scoring system processes, and calibration strategies.  

• Support AP Reader communications related to scoring platform usage, engagement portals, messaging, and operational processes.  

• Provide operational support during peak scoring periods and help resolve implementation or escalation issues related to scoring systems and workflows.  

Scoring Data Analysis & Performance Insights (20%) 

• Develop analytic models, dashboards, and reporting tools that provide real-time and post-Reading insights into scoring performance.  

• Analyze scoring trends to identify operational risks, anomalies, reliability concerns, training gaps, and workflow inefficiencies.  

• Translate complex data into actionable insights and recommendations for scoring leadership and operational stakeholders.  

• Partner with Psychometrics and Research teams to align operational metrics with scoring validity and reliability standards.  

• Establish standardized reporting and postmortem analysis processes that support continuous improvement across Reading cycles.  

Process Standardization & Continuous Improvement (15%) 

• Develop and maintain standardized processes, governance documentation, and operational frameworks for FR rating analytics and reporting.  

• Create and maintain cross-functional RACI documentation related to digital scoring platform deployment, configuration, and operational support.  

• Lead retrospective reviews following Reading cycles and synthesize findings into prioritized operational and capability improvements.  

• Drive automation and efficiency initiatives that reduce manual processes and improve operational accuracy and speed.  

• Identify and implement best practices across subjects to improve consistency in scoring operations and performance management.  

• Contribute to long-term strategy efforts supporting digital transformation and scaling of AP Reading capabilities.  

About You 

To qualify for this role, you must have: 

• Expertise in operational analytics, scoring operations, education systems, or related fields with experience supporting large-scale operational environments.  

• Strong analytical and systems-thinking skills, with demonstrated ability to translate data into operational improvements and scalable solutions.  

• Experience working with large-scale datasets and reporting tools such as Excel, SQL, BI tools, or similar technologies.  

• Strong project management and prioritization skills with the ability to manage competing priorities across multiple stakeholders.  

• Excellent written and verbal communication skills, including the ability to simplify complex analyses into actionable insights.  

• Experience collaborating across cross-functional teams including operations, technology, product, analytics, and research partners.  

• Comfort operating in both strategic planning and hands-on execution environments.  

• Experience in assessment, scoring operations, educational measurement, or education data systems strongly preferred.  

• Bachelor’s degree required; advanced degree preferred.  

• Ability to travel 2–4 times per year to College Board offices or on behalf of College Board business.  

All roles at College Board require: 

• A passion for expanding educational and career opportunities and mission-driven work.  

• Authorization to work in the United States for any employer.  

• Curiosity and enthusiasm for emerging technologies, including AI-driven solutions and digital tools.  

• Clear and concise communication skills, written and verbal.  

• A learner’s mindset and commitment to continuous growth and improvement.  

• A drive for impact and excellence through data-informed decision-making and problem-solving.  

• A collaborative and empathetic approach that fosters trust and shared success.  

About Our Process 

• Application review will begin immediately and continue until the position is filled. This role is expected to accept applications for a minimum of 5 business days.  

• While the hiring process may vary, it generally includes a recruiter screen, hiring manager interview, performance exercise, panel interview, leadership conversation, and reference checks.  

What We Offer 

At College Board, we offer more than just a paycheck—we provide a meaningful career, a supportive team, and a comprehensive package designed to help you thrive. We’re a self-sustaining nonprofit that believes in fair and competitive compensation grounded in qualifications, experience, impact, and market data. 

A Thoughtful Approach to Compensation 

• The hiring range for this role is $80,000–$130,000.  

• Compensation is based on location, experience, skills, and market benchmarks.  

• We aim to make our best offer upfront—rooted in fairness, transparency, and equity.  

• Salary structures are adjusted by geographic location to ensure fairness across markets.  

You’ll have open, transparent conversations throughout the hiring process about compensation, benefits, and what it’s like to work at College Board. 

#LI-DC1   

#LI-REMOTE