Senior Manager, Machine Learning
Senior Manager overseeing ML data operations and labeling quality at Coalition. Define guidelines and metrics, drive Label Studio platform requirements, manage vendors, and partner with ML teams to ensure high-quality labeled datasets for models.
About the job
Responsibilities
- Define annotation guidelines, taxonomies, and edge-case protocols for labeling programs. Establish gold standard datasets, IAA targets, and audit processes. Identify and remediate mislabeled data.
- Serve as primary user and requirements driver for Label Studio, defining project configurations, workflows, pre-labeling pipelines, and ML infrastructure integrations. Partner with data engineering team.
- Collaborate with ML engineers, data scientists, and product managers to translate model requirements into structured labeling tasks. Challenge ambiguous task designs.
- Source, onboard, and manage external labeling vendors and BPOs. Set quality SLAs, run calibration sessions, manage feedback loops, and hold vendors accountable for accuracy.
- Define and track metrics such as label accuracy, IAA scores, cost per label, and turnaround time. Drive improvements via active learning, model-assisted labeling, and pre-annotation.
Skills and Qualifications
- 5+ years in ML data operations, data labeling, or related field (ML engineering, data science, or data engineering with heavy labeling exposure).
- Deep understanding of annotation quality frameworks: IAA, consensus labeling, gold standard evaluation, error taxonomy, and calibration workflows.
- Direct experience managing labeling platforms (Label Studio strongly preferred; Scale AI, Labelbox, Prodigy, or similar).
- Track record managing outsourced labeling vendors or BPOs for ML data production.
- Familiarity with common ML labeling tasks: text classification, NER, document extraction, intent detection.
- Comfortable working in Python and SQL; bonus for building tooling around labeling workflows or quality measurement.
- Strong opinions on data quality and willingness to push back on poor practices.
- Experience in insurance, cybersecurity, or fintech is a plus.
Compensation
US base salary ranges from $134400/year to $200000/year depending on geographic market and factors including education, skills, experience, and location.
Skills
Label Studio, Python, SQL, Iaa, Ner, Text Classification, Document Extraction, Intent Detection, Scale Ai, Labelbox, Prodigy, Active Learning, Model-Assisted Labeling
Similar jobs
ML Engineering jobs
Axle
United States
Build and operate agentic AI systems for rare disease research, including typed workflows, evaluation, observability, and production deployment. The role requires a bachelor’s degree or equivalent experience, 5+ years building production systems, and 2+ years shipping LLM-powered applications.
6sense
United States
Build and deploy secure enterprise AI agents, integrations, and automated workflows that improve internal productivity and business operations. The role requires at least five years of enterprise, software, automation, or IT systems engineering experience, plus hands-on workflow and LLM application development.
Apartment List
United StatesBuild marketplace search and ranking features while supporting MLOps infrastructure, model deployment, feature stores, and real-time data pipelines. The role requires 5+ years of software engineering or MLOps experience, backend or full-stack expertise, and familiarity with cloud and machine learning tooling.
Blee
San Francisco, CA
Designs and ships production multi-agent compliance systems, including LLM pipelines, model training, evaluation, monitoring, and explainability. Requires 5+ years of applied AI/ML engineering experience, strong Python, and experience deploying production ML systems.
Censys
United StatesBuild and deploy production machine-learning models and data systems that classify and enrich Internet telemetry for internal platforms and customer-facing products. The role requires 5+ years of applied ML, data science, or software engineering experience, plus strong Python or Go skills.