Quality Analyst II

Key Responsibilities

Evaluate AI Quality Across the Advisor Stack

• Review pre-call briefs for accuracy, completeness and signal quality (customer context, adoption signals, retention flags)
• Evaluate in-call guidance (next-best-action prompts, suggestions and recommendation flows) for relevance and accuracy
• Score post-call outputs: quality grading, AI-generated summaries, CRM updates, sentiment signals
• Audit transcription-driven follow-ups (customer updates, recap notes, follow-up tasks) for accuracy, tone and customer impact
• Classify issues by type and severity (hallucination, missing context, wrong action, tone mismatch, KB gap) using standardized rubrics
• Participate in calibration sessions to keep scoring consistent across reviewers

Iterate Inside ORA — Light Creation Work

• Refine and iterate on prompts within existing skills and sub-agents
• Update knowledge base entries to close information gaps surfaced in reviews
• Tweak existing skills and orchestrations to improve output quality
• Test changes, validate improvements and roll out updates inside ORA
• Document what you changed and why

Surface Trends & Drive Continuous Improvement

• Tag conversations by product area, severity and issue type with consistent classification
• Maintain a clear, trusted log of issue types, trends and conversation behaviors
• Surface recurring patterns and systemic gaps to leadership and the Lead Analyst
• Recommend prioritized improvement opportunities
• Build reporting that gives Success leadership visibility into AI quality and the improvement backlog

Partner with RevOps Engineering & the Lead Analyst

• Bring structured findings, repro examples and quality criteria when a platform change is needed
• Partner with the Lead Analyst on end-to-end builds — bring evaluation signal, test the output, validate the bar
• Co-define quality criteria for new sub-agents, skills and orchestrations before they ship
• Stay close through the build to ensure improvements land as intended

Stay Sharp: Learn, Experiment, Share

• Stay current on LLM behavior, prompt patterns and AI evaluation techniques
• Run small experiments with new prompts, tools or evaluation approaches inside ORA
• Share what you learn with the broader Success team and the Lead Analyst

Requirements

• Insatiable curiosity and a self-learning mindset
• 2+ years of experience in Quality Assurance, Customer Support, AI evaluation or a related role
• Working understanding of how LLMs behave (prompt design, common failure modes, conversational AI patterns)
• General understanding of databases and how data flows between systems
• Comfort reading code and making minor edits
• Strong written communication and attention to detail
• Ability to follow structured guidelines and scoring rubrics consistently while suggesting improvements
• Comfort working with evolving tools and AI-assisted workflows
• Self-starter mentality
• Bachelor’s degree in a related field, or equivalent practical experience

Preferred Qualifications

• Hands-on experience with AI development platforms such as Claude, Cursor or similar tools
• Familiarity with retrieval-augmented generation (RAG), vector databases or knowledge-base architecture
• Familiarity with SQL or similar database querying
• Experience with APIs, webhooks or data integration between systems
• Background in customer success, support, sales enablement or call center environments
• Experience with the HighLevel platform or comparable SaaS products
• Prior experience evaluating AI-generated conversations, chatbots or automated support output