AI Operations Specialist
Operationalize AI agents for customer care by developing procedures, evaluating bot quality, analyzing performance metrics, and driving improvement with Product, Engineering, QA, and vendors. Requires 3–5 years of relevant experience, strong analytical skills, and comfort with APIs and technical documentation.
About the job
Responsibilities
AI Operating Model and SOP Development
- Design and maintain AI Operating Procedures (AOPs/SOPs) defining Agent AI behavior across supported use cases.
- Translate human-agent judgment into structured system logic, including actions, edge cases, and expected outcomes.
- Inspect and reference APIs, system actions, and integration flows in documentation.
- Update SOPs as product functionality and use cases evolve.
AI Quality Assurance and Performance Improvement
- Define and execute QA frameworks evaluating containment, accuracy, and resolution quality across vendor bots, internal bots, and LLM-based systems.
- Identify failure modes, escalation drivers, and judgment gaps.
- Partner with Product, Engineering, and vendors to improve AI performance and customer experience.
- Enable and calibrate human QA teams evaluating Agent AI at scale.
AI Analytics and Operational Insights
- Build and maintain dashboards tracking containment rate, accuracy, escalation rate, and other AI metrics.
- Analyze trends and surface actionable insights for prioritization and iteration.
- Use data to inform readiness decisions and identify performance risks.
Cross-Functional Execution
- Participate in daily standups with Product and Engineering to align on priorities and iteration loops.
- Provide structured operational feedback to vendors and internal stakeholders.
- Support release-readiness discussions and post-launch performance reviews.
- Travel internationally up to once per quarter to BPO sites supporting AI QA operations.
Requirements
- College degree or equivalent qualification.
- 3–5 years of relevant experience in operations, QA, product operations, support tooling, or an adjacent field.
- Strong problem-solving, critical-thinking, analytical, interpersonal, organizational, and customer-empathy skills.
- Ability to reason through complex systems and translate human judgment into structured processes.
- Comfort working with APIs, system actions, and technical documentation without being an engineer.
- Experience working with performance metrics and dashboards.
- Ability to lead workstreams involving Product, Engineering, QA, and external partners.
- Willingness to perform execution-focused operational work.
Compensation
- US base salary: $85,000–$100,000 annually.
- Equity and benefits included.
Skills
Ai Operations, Sop Development, Quality Assurance, APIs, LLMs, Performance Metrics, Dashboards, System Logic, Data Analysis, Customer Support, Product Operations
Similar jobs
Product Operations jobs
Sleeper
Seattle, WA
Analyzes lifecycle and product-system requirements across mobile, web, email, and in-app experiences, translating them into reusable system components, interface specifications, prototypes, and implementation guidance. The role collaborates with Product, Engineering, Data, and Marketing and validates accessibility and quality.
Harvey
New York, NYManages cross-functional legal operations projects, workflows, systems, metrics, and AI enablement for an in-house legal team. Requires 3–6 years of operations or project management experience, meaningful legal-team exposure, strong systems thinking, and familiarity with legal technology.
Bilt Rewards
New York, NY
Own the daily quality and operations loop for Bilt’s AI concierge across multiple product surfaces. The role requires strong diagnostics, cross-functional communication, experience with conversational AI, and comfort reviewing code and editing structured files.
Ramp
New York, NYOwn Juno’s product operating system and launch processes while supporting customers, investigating product issues, and translating recurring friction into actionable product insights. The role requires 3+ years in product operations, program management, support, or a related analytical customer-facing function.
Datadog
New York, NY
Own the operational infrastructure supporting Technical Solutions Enablement, including systems, knowledge management, reporting, governance, and process improvement. The role requires at least four years of operations experience and strong capabilities in scalable process design, business systems, and cross-functional project management.