# Quality Analyst II

**Company:** [GoHighLevel](https://hotfix.jobs/companies/gohighlevel)
**Location:** Remote
**Role:** QA Engineering
**Experience:** 2+ years
**Skills:** ai evaluation, Prompt Engineering, LLMs, RAG, SQL, APIs, knowledge base management, quality assurance, Customer Support, Claude, Cursor
**Posted:** 2026-07-25

> Evaluate and improve AI-powered workflows and outputs for customer advisors in ORA platform. Review pre/in/post-call AI content, classify failure modes, refine prompts and knowledge bases, surface trends, and partner with engineering on fixes. Requires 2+ years QA/support/AI eval experience, LLM understanding, and self-learning mindset.

## Job Description

## Key Responsibilities

### Evaluate AI Quality Across the Advisor Stack
- Review pre-call briefs for accuracy, completeness and signal quality (customer context, adoption signals, retention flags)
- Evaluate in-call guidance (next-best-action prompts, suggestions and recommendation flows) for relevance and accuracy
- Score post-call outputs: quality grading, AI-generated summaries, CRM updates, sentiment signals
- Audit transcription-driven follow-ups (customer updates, recap notes, follow-up tasks) for accuracy, tone and customer impact
- Classify issues by type and severity (hallucination, missing context, wrong action, tone mismatch, KB gap) using standardized rubrics
- Participate in calibration sessions to keep scoring consistent across reviewers

### Iterate Inside ORA — Light Creation Work
- Refine and iterate on prompts within existing skills and sub-agents
- Update knowledge base entries to close information gaps surfaced in reviews
- Tweak existing skills and orchestrations to improve output quality
- Test changes, validate improvements and roll out updates inside ORA
- Document what you changed and why

### Surface Trends & Drive Continuous Improvement
- Tag conversations by product area, severity and issue type with consistent classification
- Maintain a clear, trusted log of issue types, trends and conversation behaviors
- Surface recurring patterns and systemic gaps to leadership and the Lead Analyst
- Recommend prioritized improvement opportunities
- Build reporting that gives Success leadership visibility into AI quality and the improvement backlog

### Partner with RevOps Engineering & the Lead Analyst
- Bring structured findings, repro examples and quality criteria when a platform change is needed
- Partner with the Lead Analyst on end-to-end builds — bring evaluation signal, test the output, validate the bar
- Co-define quality criteria for new sub-agents, skills and orchestrations before they ship
- Stay close through the build to ensure improvements land as intended

### Stay Sharp: Learn, Experiment, Share
- Stay current on LLM behavior, prompt patterns and AI evaluation techniques
- Run small experiments with new prompts, tools or evaluation approaches inside ORA
- Share what you learn with the broader Success team and the Lead Analyst

## Requirements
- Insatiable curiosity and a self-learning mindset
- 2+ years of experience in Quality Assurance, Customer Support, AI evaluation or a related role
- Working understanding of how LLMs behave (prompt design, common failure modes, conversational AI patterns)
- General understanding of databases and how data flows between systems
- Comfort reading code and making minor edits
- Strong written communication and attention to detail
- Ability to follow structured guidelines and scoring rubrics consistently while suggesting improvements
- Comfort working with evolving tools and AI-assisted workflows
- Self-starter mentality
- Bachelor’s degree in a related field, or equivalent practical experience

## Preferred Qualifications
- Hands-on experience with AI development platforms such as Claude, Cursor or similar tools
- Familiarity with retrieval-augmented generation (RAG), vector databases or knowledge-base architecture
- Familiarity with SQL or similar database querying
- Experience with APIs, webhooks or data integration between systems
- Background in customer success, support, sales enablement or call center environments
- Experience with the HighLevel platform or comparable SaaS products
- Prior experience evaluating AI-generated conversations, chatbots or automated support output

## Similar roles

- [Product Quality Analyst](https://hotfix.jobs/jobs/235a37d0-1ff8-4fe5-b75e-569092c8ee48) - Playground - New York, NY - $60k – $75k/yr
- [Part-Time Student Worker System Test Engineer](https://hotfix.jobs/jobs/eb6e7877-4b41-4ad5-9ab5-0d9b645a8609) - Zoox - Foster City, CA
- [Software QA Engineer](https://hotfix.jobs/jobs/5ebc636a-2d5b-4c1d-b120-4b8398a3f05a) - Bevi - Boston, MA - $100k – $124k/yr
- [Quality Enginner](https://hotfix.jobs/jobs/4d4761c0-90d3-428b-b76e-ea0148b76afd) - Carbon - Sunnyvale, CA - $104k – $155k/yr
- [Product Operations & QA Specialist](https://hotfix.jobs/jobs/ddeaed75-f58d-4ece-8666-e9f9c56de33d) - AKASA - San Francisco, CA - $60k – $85k/yr

**Apply:** https://hotfix.jobs/jobs/ea8461d8-400d-4cff-ad99-5063948ade76
**Canonical:** https://hotfix.jobs/jobs/ea8461d8-400d-4cff-ad99-5063948ade76