# Assessment Designer & Learning Analyst

**Company:** [Mercor](https://hotfix.jobs/companies/mercor)
**Location:** San Francisco, CA
**Role:** Other
**Salary:** $120k – $160k/yr
**Experience:** 1+ years
**Skills:** Psychometrics, Classical Test Theory, Item Response Theory, Item Analysis, Validity Studies, Reliability Analysis, Quantitative Research Methods, Python, R, Stata, Excel, Google Sheets, Regression Analysis, Predictive Modeling, Mixed Methods Research
**Posted:** 2026-06-30

> Design rigorous assessments and certification frameworks to measure expert skills for AI training projects. Conduct learning analytics, item-level analysis, validity studies, and mixed-methods research to link assessments to real-world performance and drive program improvements.

## Job Description

## Assessment Design
- Design and continuously improve assessments and certification frameworks that validly and reliably measure expert readiness for specific project types.
- Build assessments and measurements of skills that are consistent, interpretable, and predictive of on-the-job performance.
- Develop item banks, scoring guides, and inter-rater reliability protocols for evaluating complex human judgment tasks.
- Run validity studies to ensure assessments measure what they intend to measure.

## Learning Analytics & Impact Analysis
- Analyze the relationship between instructional materials, assessments, and expert performance; identify what's working and make recommendations.
- Perform item-level analysis (difficulty, discrimination, reliability) and iterate based on findings.
- Investigate relationships between assessment performance and real-world expert effectiveness.
- Build reports and dashboards that surface actionable insights to program and operations teams.
- Design and analyze quasi-experimental, quantitative, and qualitative (mixed methods) studies to understand effective interventions.

## Ongoing Measurement & Improvement
- Track certification and assessment outcomes over time and flag when programs need revision.
- Partner with learning designers and project teams to translate findings into program improvements.
- Maintain a continuous improvement mindset: ship, measure, learn, iterate.

## Similar roles

- [Field Marketing Coordinator](https://hotfix.jobs/jobs/a8a93a62-9ba0-4bed-a3a4-4d98dac7e6c1) - PlanetScale - San Francisco, CA - $120k – $140k/yr
- [Learning Designer](https://hotfix.jobs/jobs/ee45c091-de6d-46c8-ade8-0864e03ff3c0) - Mercor - San Francisco, CA - $125k – $125k/yr
- [Commercial Energy Development Associate](https://hotfix.jobs/jobs/2f9f1825-7115-4382-9434-efb9e43d5a55) - Crusoe - San Francisco, CA - $125k – $150k/yr
- [Future AI Founder](https://hotfix.jobs/jobs/3bce78cd-5d1d-4014-91ee-bd5560b2d785) - Harper - San Francisco, CA - $125k – $170k/yr
- [Predoctoral Young Investigator, Robotics](https://hotfix.jobs/jobs/155869fd-70df-40aa-bdb6-0b1d61262995) - Ai2 - Seattle, WA - $113k – $113k/yr

**Apply:** https://hotfix.jobs/jobs/215b2afd-1ba3-4186-8c4a-07090e01c28b
**Canonical:** https://hotfix.jobs/jobs/215b2afd-1ba3-4186-8c4a-07090e01c28b