Software Quality Engineer
The Software Quality Engineer will define QA strategies, build automated test frameworks, and validate AI agents and LLM-powered applications. The role requires 5+ years of quality engineering experience, strong Python or TypeScript skills, GitHub Actions, Playwright, and familiarity with AI/ML testing.
About the job
Responsibilities
- Define and implement quality assurance strategies and test plans for AI agents and LLM-powered applications.
- Design, develop, and maintain robust, scalable automated test frameworks using Python or TypeScript and modern testing tools.
- Collaborate with product managers, machine learning engineers, and data scientists to create test cases and validation criteria for complex AI features and model behavior.
- Improve testing processes and infrastructure, integrating automated checks into CI/CD pipelines.
- Identify, document, track, and prioritize software defects; perform root-cause analysis and debug logs to provide actionable feedback.
- Monitor production systems and AI model performance, contributing to post-release quality validation.
- Establish and maintain test-data strategies and mock services for stable, isolated, repeatable test execution.
- Design, develop, or integrate agentic AI systems, AI skills, and the Model Context Protocol (MCP).
- Champion quality practices across engineering teams and manage the full defect lifecycle.
Requirements
- 5+ years of hands-on experience in software quality assurance or engineering, focused on complex distributed systems or AI/ML applications.
- 5+ years of experience working with GitHub Actions.
- Proficiency in Python or TypeScript for test automation.
- Experience with modern testing frameworks such as Playwright.
- Strong understanding of AI/ML concepts, including model evaluation metrics, data pipelines, and the challenges of testing generative AI outputs.
- Strong analytical, problem-solving, communication, and collaboration skills.
- Ability to drive initiatives independently in a fast-paced startup environment.
Nice-to-haves
- Experience with AWS, Azure, or Google Cloud.
- Experience with Docker and Kubernetes in a CI/CD environment.
Compensation and Benefits
- Competitive compensation and company stock options.
- Generous paid time off and company holidays.
- Medical and dental insurance.
- 16 weeks of paid parental leave for all parents.
- Fertility and family planning support.
- Early-detection cancer testing.
- Competitive pension scheme and company contribution.
- Wellness, learning and development stipends.
- Company-wide and team off-sites.
Skills
Python, TypeScript, Playwright, GitHub Actions, CI/CD, AI/ML, Model Evaluation, Data Pipelines, AWS, Azure, GCP, Docker, Kubernetes, Model Context Protocol
Similar jobs
QA Engineering jobs
Circle
Remote
Senior Quality Engineer partnering across product teams to assess risk, build web and API test automation, investigate technical failures, and improve quality practices. Requires strong Playwright and TypeScript experience, exploratory testing skills, CI/CD knowledge, and the ability to debug application code.
Encord
London, United Kingdom
Own Encord’s data-quality standards, automated evaluation systems, and audit operations for human-generated AI data. The role combines hands-on Python and SQL development with statistical quality measurement and leadership of a distributed audit team.
Spotify
New York, NYEvaluates and defines quality standards for Spotify’s AI voice models across products, languages, and use cases. The role designs qualitative and quantitative evaluation programs, benchmarks competing technologies, and translates findings into product and research recommendations.
Hudl
London, United Kingdom
Leads quality strategy and hands-on engineering for a shared MLOps platform, building test frameworks, deployment safety tooling, and observability systems. The role requires strong platform engineering expertise, technical leadership, and experience guiding teams across complex ML systems.