# Senior Machine Learning Engineer, Developer Advocacy | US | Remote

**Company:** [Grafana Labs](https://hotfix.jobs/companies/grafana-labs)
**Location:** Remote
**Role:** ML Engineering
**Salary:** $154k – $185k/yr
**Experience:** 5+ years
**Skills:** Recommendation Systems, personalization, ranking, next best action, Model Training, model monitoring, model validation, experiment design, A/B Testing, feature engineering, Go, TypeScript, gRPC, http, Distributed Systems
**Posted:** 2026-07-22

> Build, deploy, and iterate on personalized real-time recommendation models for Grafana's Interactive Learning system to help users discover relevant guides and product experiences. Requires strong experience in recommendation/personalization science, applied ML model ownership in production, and distributed systems (Go/TS/gRPC).

## Job Description

## What You’ll Be Doing

Evolve the Interactive Learning Plugin's recommendation system
- Develop increasingly personalized approaches to candidate selection, ranking, sequencing, and next-best-action recommendations.

Own a real-time recommendation service
- Build and operate applied models: Develop, validate, version, monitor, and iterate on models used by the recommendation system.
- Own model training & serving.

Define what recommendation quality means
- Develop offline, online, and longitudinal measures of recommendation performance.
- Own feature pipelines, monitoring of the model and architecture.

Ship incremental improvements
- Use the data and infrastructure available today while identifying the instrumentation and platform capabilities needed tomorrow.
- Integrate improvements into the existing recommender rather than waiting for a complete replacement system.

Partner across disciplines
- Work closely with software engineers & data analysts to productionize models and integrate them safely into the recommender service.
- Partner with the Product Analytics team on metric definitions, instrumentation, data quality, dashboards, and experiment analysis.
- Collaborate with Developer Advocacy, Docs, Product, Engineering, GTM, and other teams to translate ambiguous needs into testable hypotheses and measurable product decisions.
- Explain modeling choices, tradeoffs, uncertainty, and results clearly to both technical and non-technical audiences.

## What Makes You a Great Fit

Strong candidates should demonstrate credible ability across all three core areas below and be particularly strong in at least two.

**Recommendation and personalization science**: you have built recommendation, ranking, search, matching, propensity, or next-best-action systems. You are comfortable beginning with simple, explainable approaches when they are the best way to learn.

**HTTP/gRPC, streaming, Go/TypeScript**: previous experience in distributed systems.

**Applied model ownership**: You have personally built, validated, monitored, and iterated on models used in a product or operational environment. You can work effectively in version-controlled codebases and collaborate with engineers on production implementation.

You should also be a strong product thinker and technical communicator. You can take an ambitious and ambiguous objective, identify the most important unknowns, and create a sequence of models and experiments that steadily improves the product.

## Bonus Points For

- Experience with content, education, onboarding, or learning recommendation systems
- Experience with SaaS product telemetry and customer-account data
- Experience using warehouse-scale behavioral data
- Experience with directed graphs, sequence models, or prerequisite-aware recommendations
- Experience with contextual bandits or other exploration strategies
- Familiarity with Grafana or the broader observability ecosystem
- Experience with open source software or transparent development practices
- Experience working with privacy, fairness, explainability, or responsible personalization constraints

## Compensation & Rewards

In the United States, the base compensation range for this role is $154,445 - $185,334. All of our roles include Restricted Stock Units (RSUs), giving every team member ownership in Grafana Labs' success.

## Similar roles

- [Senior AI Engineer](https://hotfix.jobs/jobs/fdf9ce54-1a65-4442-a904-d175b0df9243) - Grafana Labs - Remote - $154k – $185k/yr
- [Senior Software Engineer - AI and Automation, Data & Analytics | USA | Remote](https://hotfix.jobs/jobs/d182728b-ee48-41c1-8d1f-b440bbbc7c0c) - Grafana Labs - Remote - $154k – $185k/yr
- [Sr. Software Engineer, Machine Learning](https://hotfix.jobs/jobs/19067112-c9ce-4c49-8706-9f960d168a9c) - Pinterest - Remote - $156k – $320k/yr
- [Sr. Software Engineer, tvScientific](https://hotfix.jobs/jobs/82f201f6-0d12-4205-9a11-8a912b57fef9) - Pinterest - Remote - $156k – $320k/yr
- [Senior Software Engineer - ML Infrastructure](https://hotfix.jobs/jobs/ce708525-34f1-4474-9516-673a3457e146) - Applied Intuition - Sunnyvale, CA - $153k – $222k/yr

**Apply:** https://hotfix.jobs/jobs/0b4d788e-02f5-4f9a-8fa9-684865579ed9
**Canonical:** https://hotfix.jobs/jobs/0b4d788e-02f5-4f9a-8fa9-684865579ed9