# Staff Machine Learning Engineer, Generative AI

**Company:** [Okta](https://hotfix.jobs/companies/okta)
**Location:** Toronto, Canada
**Role:** ML Engineering
**Salary:** CA$168k – CA$231k/yr
**Experience:** 7+ years
**Skills:** Python, Go, TypeScript, Machine Learning, Aws Bedrock, OpenAI, Anthropic, RAG, Embeddings, Litellm, LangGraph, LangChain, Llamaindex, Mcp, PyTorch
**Posted:** 2026-08-18

> The Staff Machine Learning Engineer will architect and deploy scalable generative AI and machine learning systems, including retrieval, inference, evaluation, and agentic workflows. The role requires 7+ years of software development experience, strong Python skills, applied ML expertise, and deep familiarity with modern GenAI platforms and frameworks.

## Job Description

## Responsibilities
- Architect, design, and deploy robust machine learning and generative AI systems integrated with platform services.
- Establish scalable LLMOps pipelines for production.
- Drive technical decisions balancing simplicity, flexibility, reliability, and performance.
- Tune, optimize, and deploy agentic applications with a focus on performance, reliability, and security.
- Partner with Product, Security, and Platform Engineering teams to design innovative, trustworthy AI-powered experiences.
- Design and implement scalable infrastructure and platform services for large-scale generative AI use cases.
- Collaborate with product managers, researchers, and engineers to deliver secure, high-quality AI/ML systems.
- Design observable ML and generative AI systems integrating retrieval, inference, and evaluation pipelines.
- Develop structured prompting, context retrieval, and RAG workflows for Claude-based systems.
- Build automated evaluation pipelines measuring model quality, correctness, groundedness, and safety.
- Implement schema validation, structured output enforcement, and guardrails for reliable, auditable, compliant AI outputs.
- Mentor and coach engineers.

## Requirements
- 7+ years of software development experience.
- Strong Python programming expertise; familiarity with Go or TypeScript is a plus.
- Hands-on applied machine learning experience, including feature engineering, model training, and fine-tuning.
- Experience with generative AI platforms such as AWS Bedrock, OpenAI, and Anthropic.
- Deep understanding of retrieval-augmented generation, embeddings, and knowledge-base workflows.
- Experience with LiteLLM, LangGraph, LangChain, LlamaIndex, MCP, or related AI agent frameworks.
- Familiarity with FastAPI, PyTorch, TensorFlow, Spark ML, and workflow orchestration tools such as Airflow.
- Experience defining evaluation metrics, pipelines, and feedback loops for ML and generative AI systems.
- Ability to collaborate with product and engineering teams, lead greenfield initiatives, navigate ambiguity, and iterate quickly.
- Experience building tools or infrastructure for AI/ML applications and understanding AI-native developer lifecycles.

## Nice to haves
- Experience integrating AI-driven systems with identity, authentication, or security products.
- Exposure to ethical AI, model risk, or compliance frameworks.
- Familiarity with evaluation datasets, synthetic data generation, or LLM-as-a-judge methods.

## Compensation and benefits
- Annual base salary range in Canada: **$168,000–$231,000 CAD**.
- Equity, where applicable, bonus, health, dental, and vision insurance.
- RRSP with a match, healthcare spending, telemedicine, and paid leave, including PTO and parental leave.

## Similar jobs

- [Staff AI Engineer](https://hotfix.jobs/jobs/ef3edbe2-3c9e-4733-8345-40c8d0307d13) - Grafana Labs - Remote - CA$164k – CA$197k/yr
- [Staff Machine Learning Engineer](https://hotfix.jobs/jobs/2cb6da8f-4a54-448d-a4b6-fc05f1412a12) - Payabli - Remote
- [Staff Backend Engineer – AI](https://hotfix.jobs/jobs/ec44131d-5167-49f2-bdf5-b581d146fa56) - Stream - Amsterdam, Netherlands
- [Staff AI Marketing Systems Engineer](https://hotfix.jobs/jobs/a946d08e-9494-4af3-9eb4-f737c10d0543) - 1Password - Remote
- [Senior Machine Learning Engineer, Economist](https://hotfix.jobs/jobs/be808d21-6efa-4349-9efe-087188c17002) - Instacart - Remote - $173k – $219k/yr

**Apply:** https://hotfix.jobs/jobs/21cf0448-14c7-40b0-86b0-591f6da62b11
**Canonical:** https://hotfix.jobs/jobs/21cf0448-14c7-40b0-86b0-591f6da62b11