Skip to content
MercorMercorSan Francisco, CA

Assessment Designer & Learning Analyst

Design rigorous assessments and certification frameworks to measure expert skills for AI training projects. Conduct learning analytics, item-level analysis, validity studies, and mixed-methods research to link assessments to real-world performance and drive program improvements.

120k – 160k/yr
On-site1+ YOEOther

About the role

Assessment Design

  • Design and continuously improve assessments and certification frameworks that validly and reliably measure expert readiness for specific project types.
  • Build assessments and measurements of skills that are consistent, interpretable, and predictive of on-the-job performance.
  • Develop item banks, scoring guides, and inter-rater reliability protocols for evaluating complex human judgment tasks.
  • Run validity studies to ensure assessments measure what they intend to measure.

Learning Analytics & Impact Analysis

  • Analyze the relationship between instructional materials, assessments, and expert performance; identify what's working and make recommendations.
  • Perform item-level analysis (difficulty, discrimination, reliability) and iterate based on findings.
  • Investigate relationships between assessment performance and real-world expert effectiveness.
  • Build reports and dashboards that surface actionable insights to program and operations teams.
  • Design and analyze quasi-experimental, quantitative, and qualitative (mixed methods) studies to understand effective interventions.

Ongoing Measurement & Improvement

  • Track certification and assessment outcomes over time and flag when programs need revision.
  • Partner with learning designers and project teams to translate findings into program improvements.
  • Maintain a continuous improvement mindset: ship, measure, learn, iterate.

Skills

PsychometricsClassical Test TheoryItem Response TheoryItem AnalysisValidity StudiesReliability AnalysisQuantitative Research MethodsPythonRStataExcelGoogle SheetsRegression AnalysisPredictive ModelingMixed Methods Research

Similar roles

PlanetScale

Field Marketing Coordinator

PlanetScaleSan Francisco, CA

Coordinate in-person events including executive dinners, conferences, meetups, and community activations to connect developers and tech leaders with PlanetScale. Requires 2+ years in event or field marketing with strong organizational and project management skills.

120k – 140k/yr
On-site2+ YOEOther
Mercor

Learning Designer

MercorSan Francisco, CA

Design instructional infrastructure and training programs for internal teams and talent experts executing AI data annotation and evaluation tasks. Requires 2+ years in learning/instructional design with strong writing and pedagogical skills.

125k – 125k/yr
On-site2+ YOEOther
Crusoe

Commercial Energy Development Associate

CrusoeSan Francisco, CA +1

Supports development of GW-scale energy projects for AI compute campuses through pipeline sourcing, partner engagement, due diligence, deal execution, market analysis, and cross-functional collaboration. Requires 4+ years in energy/real estate, power marketing experience, and relevant degree.

125k – 150k/yr
On-site4+ YOEOther
Harper

Future AI Founder

HarperSan Francisco, CA

Embeds in operations to handle sales, applications, and customer service while documenting processes and edge cases to build scalable AI systems and automations. Targets exceptional early-career talent with high agency for intensive founder-like training.

125k – 170k/yr
On-siteEntry levelOther
Ai2

Predoctoral Young Investigator, Robotics

Ai2Seattle, WA

Early-career researcher on robotics team investigating foundation models for robot manipulation, skill learning, and sim-to-real transfer in scalable simulations. Prepares candidates for PhD programs through mentorship, publications, and open-source contributions. Requires bachelor’s/master’s in relevant field.

113k – 113k/yr
On-siteEntry levelOther