Senior Manager, Machine Learning

Responsibilities

• Define annotation guidelines, taxonomies, and edge-case protocols for labeling programs. Establish gold standard datasets, IAA targets, and audit processes. Identify and remediate mislabeled data.
• Serve as primary user and requirements driver for Label Studio, defining project configurations, workflows, pre-labeling pipelines, and ML infrastructure integrations. Partner with data engineering team.
• Collaborate with ML engineers, data scientists, and product managers to translate model requirements into structured labeling tasks. Challenge ambiguous task designs.
• Source, onboard, and manage external labeling vendors and BPOs. Set quality SLAs, run calibration sessions, manage feedback loops, and hold vendors accountable for accuracy.
• Define and track metrics such as label accuracy, IAA scores, cost per label, and turnaround time. Drive improvements via active learning, model-assisted labeling, and pre-annotation.

Skills and Qualifications

• 5+ years in ML data operations, data labeling, or related field (ML engineering, data science, or data engineering with heavy labeling exposure).
• Deep understanding of annotation quality frameworks: IAA, consensus labeling, gold standard evaluation, error taxonomy, and calibration workflows.
• Direct experience managing labeling platforms (Label Studio strongly preferred; Scale AI, Labelbox, Prodigy, or similar).
• Track record managing outsourced labeling vendors or BPOs for ML data production.
• Familiarity with common ML labeling tasks: text classification, NER, document extraction, intent detection.
• Comfortable working in Python and SQL; bonus for building tooling around labeling workflows or quality measurement.
• Strong opinions on data quality and willingness to push back on poor practices.
• Experience in insurance, cybersecurity, or fintech is a plus.

Compensation

US base salary ranges from $134400/year to $200000/year depending on geographic market and factors including education, skills, experience, and location.