Member of Technical Staff, North Modelling
Own North’s evaluation strategy and build feedback systems that connect real enterprise workflows, user needs, and production failures to model improvements. The role requires strong applied machine learning judgment and the ability to translate product signals into rigorous, actionable evaluations.
About the job
Responsibilities
- Own the evaluation strategy for North across agent workflows, tool use, enterprise knowledge work, deep research, document creation and editing, and other human-AI interactions.
- Build high-quality evaluations based on user feedback, production failures, privacy-preserving usage logs, internal dogfooding, customer needs, and product goals.
- Create systems that continuously turn product, customer, and user learnings into evaluations.
- Translate evaluation results, customer-derived data, and product context into recommendations for modelling teams.
- Identify model failures, capability gaps, and North-specific needs to guide model selection, patches, and regular model updates.
Requirements
- Experience improving LLM-powered, agent-powered, or AI-product systems through evaluations, feedback loops, data curation, prompting, model adaptation, or model selection.
- Strong applied machine learning engineering judgment.
- Ability to reason about model behavior, evaluation validity, product outcomes, and production tradeoffs.
- Experience working with users and product teams and translating qualitative signals into actionable measurement.
- Self-directed and practical approach to open-ended technical and product problems.
- Interest in making AI systems useful in real enterprise settings.
Nice-to-haves
- Experience with representative tasks, precise rubrics, clean data, failure analysis, regression tracking, and identifying misleading metrics.
- Experience evaluating agentic workflows and enterprise knowledge work.
Compensation & Benefits
- Weekly lunch stipend of $75/£75 or equivalent in local currency.
- Full health and dental benefits, including a mental health budget.
- RRSP matching, 401K, or pension scheme.
- Parental leave top-up for up to six months for either parent.
- Annual enrichment benefits for arts and culture, fitness and wellness, quality time, and workspace improvements.
- Education and learning stipend for conferences, courses, and coaching.
- Six weeks of paid vacation.
- Travel budget for other offices and an annual company offsite.
- Coworking benefit and a $500 home office stipend.
Skills
LLMs, Machine Learning, Machine Learning Engineering, AI Agents, Model Evaluation, Evaluation Design, Feedback Loops, Data Curation, Prompting, Model Adaptation, Model Selection, Failure Analysis, Regression Tracking, Privacy-Preserving Logs, Enterprise Ai
Similar jobs
ML Engineering jobs
Rollstack
United StatesBuild production AI capabilities for automated slide and document generation, working across LLM applications, data analysis, and content generation. The role requires 3+ years in machine learning and NLP, advanced Python, and experience with LLM frameworks and production systems.
OpenAI
London, United Kingdom
Build and deploy AI-powered products for digital-native customers, taking systems from experimentation through production and scale. The role requires strong Python skills, hands-on production engineering, systematic AI evaluation, and the ability to navigate reliability, security, governance, and customer impact.
Coalition Security
United StatesBuild and operate LLM-powered agents that automate business workflows, integrating systems of record through MCP and validating behavior with structured evaluations. Requires 5+ years of software development experience, strong Python and SQL, and hands-on experience shipping agents to production.
Elliptic
London, United Kingdom
Build full-stack AI agent fleets, APIs, workflows, and internal services that automate complex business processes. The role requires at least five years of engineering experience, hands-on LLM framework experience, production AWS expertise, Kubernetes, and strong API and database skills.
Protege
Remote
Build the technical foundation for a new business vertical, creating reusable infrastructure and leading early customer engagements from scoping through delivery. The role requires 3+ years of engineering experience, strong Python and SQL skills, backend/data expertise, and comfort operating in ambiguity.