AI Data Analyst

Responsibilities

• Conduct ongoing human labeling for priority AI workflows, including response quality, task success, MCP and tool use, and end-to-end workflows.
• Triage bad-query reports, downvotes, escalations, and routed quality issues; categorize failure modes and provide analysis to partner teams.
• Perform qualitative analysis of hallucinations, retrieval failures, tool-use issues, weak grounding, and poor workflow completion.
• Follow and improve labeling guidelines and rubrics for consistent, realistic judgments.
• Maintain labeled datasets, golden sets, and regression slices; monitor coverage, drift, leakage, and difficulty distribution.
• Participate in grader calibration and validate LLM-as-a-judge outputs against human labels.
• Partner with Engineering, Product, QA, and evaluation owners on quality reads, benchmark updates, and release-readiness decisions.
• Contribute to recurring quality reporting with failure modes, coverage gaps, quality shifts, and recommendations.

Requirements

• 3–5 years of experience in data labeling, data analysis, QA, or a related field.
• Strong analytical judgment and attention to detail.
• Experience evaluating AI-generated outputs or working with NLP, search, recommendation, or other ML systems.
• Familiarity with structured qualitative analysis, basic SQL/data retrieval, and spreadsheet workflows.
• Clear written and verbal communication skills.
• Ability to work independently in ambiguous, fast-moving environments and collaborate across functions.

Compensation & Benefits

• Compensation is determined by location, level, job-related knowledge, skills, and experience.
• Certain roles may be eligible for variable compensation, equity, and benefits.
• Hybrid role requiring four days per week in the office.