# Staff AI Engineer - Grafana AI/ML

**Company:** [Grafana Labs](https://hotfix.jobs/companies/grafana-labs)
**Location:** Remote
**Role:** ML Engineering
**Salary:** CA$186k – CA$230k/yr
**Experience:** 7+ years
**Skills:** LLMs, Prompt Engineering, Generative AI, Agent Frameworks, Multi-Agent Workflows, AWS, GCP, Microsoft Azure, Observability, Kubernetes, Docker, Terraform, Model Fine-Tuning, Backend Engineering, Full-Stack Engineering
**Posted:** 2026-08-04

> Build and ship AI-powered features for incident detection, triage, resolution, and observability workflows. The role requires strong production software engineering experience, practical LLM and GenAI expertise, cloud-native exposure, and a pragmatic approach to rapid experimentation.

## Job Description

## Responsibilities
- Build and deliver high-performance AI features that help users detect, triage, and resolve incidents using observability data and tools.
- Prototype, test, ship, and evolve LLM- and agent-powered workflows for incident lifecycle management and automated analysis.
- Collaborate with data analysts, product managers, and designers on AI-driven product features.
- Integrate agentic components with internal tools, alerting systems, runbooks, and developer workflows.
- Use AI and automation tools to improve product functionality and development workflows.
- Own AI solutions end to end, ensuring they are innovative, scalable, maintainable, and aligned with user workflows.

## Requirements
- Strong experience building production software systems, particularly backend and/or full-stack systems.
- Experience with LLMs, prompt engineering, and GenAI-powered applications.
- Proven delivery of production software actively used by customers.
- Exposure to cloud-native environments such as AWS, GCP, or Azure.
- Experience using observability tools to understand and troubleshoot system behavior.
- Ability to iterate quickly, work through ambiguity, take initiative, and communicate effectively across teams.

## Nice-to-haves
- Experience with agent frameworks or multi-agent workflows.
- Experience with infrastructure or DevOps tooling such as Kubernetes, Docker, and Terraform.
- Familiarity with model fine-tuning techniques.
- Experience building observability tooling.

## Compensation and Benefits
- Base compensation in Canada: CAD 186,368–CAD 230,000.
- Restricted Stock Units (RSUs) are included with all roles.
- Company-funded usage budget for modern AI coding assistants.
- Access to frontier models from OpenAI, Anthropic, and Google.

## Similar jobs

- [Staff Machine Learning Engineer, Generative AI](https://hotfix.jobs/jobs/21cf0448-14c7-40b0-86b0-591f6da62b11) - Okta - Toronto, Canada - CA$168k – CA$231k/yr
- [Staff AI Engineer](https://hotfix.jobs/jobs/ef3edbe2-3c9e-4733-8345-40c8d0307d13) - Grafana Labs - Remote - CA$164k – CA$197k/yr
- [Staff Machine Learning Engineer](https://hotfix.jobs/jobs/2cb6da8f-4a54-448d-a4b6-fc05f1412a12) - Payabli - Remote
- [Staff Backend Engineer – AI](https://hotfix.jobs/jobs/ec44131d-5167-49f2-bdf5-b581d146fa56) - Stream - Amsterdam, Netherlands
- [Staff AI Marketing Systems Engineer](https://hotfix.jobs/jobs/a946d08e-9494-4af3-9eb4-f737c10d0543) - 1Password - Remote

**Apply:** https://hotfix.jobs/jobs/e65de56a-140e-4f25-994c-fba8b17eebed
**Canonical:** https://hotfix.jobs/jobs/e65de56a-140e-4f25-994c-fba8b17eebed