# Staff AI Engineer

**Company:** [Grafana Labs](https://hotfix.jobs/companies/grafana-labs)
**Location:** Remote
**Role:** ML Engineering
**Salary:** CA$164k – CA$197k/yr
**Experience:** 8+ years
**Skills:** Python, JavaScript, Node.js, LangChain, Crewai, Anthropic Mcp, BigQuery, GCP, Cloud Functions, Cloud Run, RAG, n8n, Workato, Vector Databases, React
**Posted:** 2026-08-11

> Builds and owns production multi-agent AI infrastructure, backend integrations, and workflow automation for marketing operations. Requires 8+ years of software engineering experience, strong Python and JavaScript/Node.js skills, production LLM experience, and deep Google Cloud expertise.

## Job Description

## Responsibilities

### Agentic Systems & AI Infrastructure
- Own end-to-end development of multi-agent AI systems, including architecture, implementation, testing, deployment, and ongoing operation.
- Build modular, composable agentic systems using orchestration frameworks such as LangChain, CrewAI, and Anthropic MCP.
- Develop reusable agentic skills for Slack, dashboards, internal applications, and CLIs.
- Implement observability and feedback loops, including logging, performance metrics, prompt iteration, model evaluation, and cost management.
- Establish governance and compliance standards for AI workflows, including access controls, audit trails, PII handling, and human-in-the-loop escalation paths.

### Systems Integration & Backend Services
- Build MCP servers, APIs, CLIs, and microservices connecting AI models to BigQuery, Slack, CRMs, email, calendars, and analytics tools.
- Architect retrieval-augmented generation (RAG) data flows connecting LLMs to internal knowledge bases, customer data, and real-time business context.
- Build serverless or containerized services with GCP Cloud Functions and Cloud Run.

### Automation & Workflow Enablement
- Partner with RevOps, Demand Generation, Regional Marketing, and SDR teams to identify high-impact automation opportunities and deliver measurable business outcomes.
- Design and deploy workflows using n8n, Workato, or custom platforms with CI/CD, testing, and production reliability standards.
- Build self-service systems with documentation, playbooks, and enablement materials.

## Requirements
- 8+ years of software engineering experience, with depth in backend development, systems integration, or data/analytics engineering.
- 2+ years of hands-on experience applying LLMs or AI to production workflows.
- Strong proficiency in Python and JavaScript/Node.js.
- Experience with Git-based workflows, code review, and testing.
- Experience with prompt engineering, RAG, function calling/tool use, structured output parsing, and evaluation.
- Experience building and operating multi-agent systems at scale, including decomposition, orchestration, state management, and production monitoring.
- Deep familiarity with Google Cloud Platform, BigQuery, Cloud Functions, and Cloud Run.
- Understanding of LLM failure modes and production mitigations, including confidence thresholds, fallback logic, human escalation, and cost/latency management.
- Ability to identify high-leverage problems and deliver end-to-end with minimal direction.
- Familiarity with AI-assisted development tools such as GitHub Copilot, Cursor, and Claude Code.
- Clear technical communication skills for engineering and business audiences.

## Nice-to-Haves
- Experience with vector databases or retrieval pipelines, including Pinecone, Weaviate, ChromaDB, Qdrant, or pgvector.
- Familiarity with Salesforce, Customer.io, HubSpot, Marketo, or Outreach.
- Experience with React or Slack Block Kit.
- Experience with LangSmith, Weights & Biases, or custom AI evaluation frameworks.
- Experience with n8n, Temporal, Prefect, or Airflow.
- Familiarity with Model Context Protocol (MCP).
- Experience automating marketing, sales, or customer success workflows in a B2B SaaS environment.
- Active participation in open-source communities.

## Compensation
- Base compensation in Canada: **CAD 164,490–CAD 197,389**.
- Roles include Restricted Stock Units (RSUs).

## Similar jobs

- [Staff Machine Learning Engineer, Generative AI](https://hotfix.jobs/jobs/21cf0448-14c7-40b0-86b0-591f6da62b11) - Okta - Toronto, Canada - CA$168k – CA$231k/yr
- [Staff Applied Scientist, Personalization](https://hotfix.jobs/jobs/f3f16be4-af38-4045-b3a6-cb954a5272da) - OnePay - Remote - $180k – $250k/yr
- [Staff AI Enablement Engineer](https://hotfix.jobs/jobs/f66b54fc-c0f6-41bc-aa3e-dd03ad31f295) - Talkiatry - Remote - $190k – $230k/yr
- [Senior/Staff Engineer, Machine Learning - Online Mapping](https://hotfix.jobs/jobs/2dbc058c-d88d-4b51-8d5e-acacb5130b4c) - Nuro - Mountain View, CA - $194k – $291k/yr
- [Senior/Staff Software Engineer, ML Inference Platform](https://hotfix.jobs/jobs/1b014e68-d8bc-4d84-b03a-d94b7a57c90f) - Nuro - Mountain View, CA - $194k – $352k/yr

**Apply:** https://hotfix.jobs/jobs/ef3edbe2-3c9e-4733-8345-40c8d0307d13
**Canonical:** https://hotfix.jobs/jobs/ef3edbe2-3c9e-4733-8345-40c8d0307d13