# Staff Backend Engineer - Application Core Services, Stacks | USA | Remote

**Company:** [Grafana Labs](https://hotfix.jobs/companies/grafana-labs)
**Location:** Remote
**Role:** Backend Engineering
**Skills:** Go, Kubernetes, AWS, Azure, GCP, Grafana, Mimir, Loki, Tempo, Microservices, Reconciliation Systems, Stateful Systems, Eventual Consistency
**Posted:** 2026-04-30

> Builds and operates backend reconciliation systems for Grafana Cloud stack lifecycle management, ensuring state alignment across services like SSS, grafana.com, and infrastructure. Collaborates cross-team on reliability, scalability, and operational improvements for customer stacks.

## Job Description

## Responsibilities
- Design, build, and operate reconciliation systems, including the SSS backend, to track desired stack state, detect and repair drift across stack templates, grafana.com state, Hosted Grafana, and actual customer stack configuration.
- Collaborate across SSS, grafana.com, and deployment configurations to ensure stack lifecycle workflows remain reliable, observable, and resilient.
- Improve operational efficiency by reducing deployment complexity (e.g., aiming for single PR regional SSS deployment) and contributing to the Stack Config Reconciliation project.
- Manage rollout mechanisms for provisioned plugins, dashboards, data sources, Grafana versions, release channels, and stack-level configuration.
- Support new region and cluster rollouts, including the operational paths required to bring stacks online safely in new Grafana Cloud regions.
- Improve incident response and recovery paths for stack misalignment, reconciliation failures, plugin rollout issues, and Hosted Grafana integration failures.
- Partner with Product, Hosted Grafana, Infrastructure, Support, and adjacent AppCore squads on customer-impacting stack lifecycle work.
- Contribute to roadmap planning, technical design, OnCall improvements, and long-term simplification of stack operations.
- Own production behavior of systems, including improving runbooks, dashboards, alerts, reconciliation safety, rollout controls, and recovery procedures.
- Debug across service boundaries and make careful changes in systems affecting customer stacks.

## Requirements
- Thrive on building production systems where correctness, scalability, and operational clarity are paramount.
- Comfortable collaborating asynchronously across time zones and taking full ownership of critical systems.
- Lead major projects, coordinate across service boundaries, and help define technical direction.
- Solve challenges related to stateful systems, eventual consistency, and reconciliation loops.
- Transform ambiguous lifecycle requirements into explicit, modular solutions.
- Break down complex systems work into safe, iterative increments.
- Communicate technical tradeoffs to internal stakeholders and adjacent product teams.
- Write efficient, readable, and easy-to-maintain code.
- Comfortable with on-call responsibilities (balanced globally, ~12 daylight hours per day).
- Experience with AI-assisted development practices (e.g., GPT, Claude, Gemini models for prototyping, testing, refactoring).

## Similar roles

- [Staff Software Engineer, Backend](https://hotfix.jobs/jobs/807748ff-c0bb-45b6-8d08-95df45fd7a54) - Rippling - San Francisco, CA - $189k – $315k/yr
- [Staff Backend Engineer](https://hotfix.jobs/jobs/b356996c-9ea9-4c66-8774-0cf5944dad09) - Aurelian - Seattle, WA - $200k – $300k/yr
- [Staff Backend Engineer, Identity Threat Protection](https://hotfix.jobs/jobs/f26dd822-f888-4c9a-8336-09762587d7cc) - Okta - San Francisco, CA - $194k – $267k/yr
- [Staff Software Engineer, Platform](https://hotfix.jobs/jobs/b2d42759-cf0d-4deb-ac34-f042721e13b6) - Zus Health - Remote - $170k – $200k/yr
- [Senior Staff Software Engineer, Payments](https://hotfix.jobs/jobs/be4bb7e7-7e74-474b-8f37-260b2a1cb607) - Coinbase - Remote - $254k – $299k/yr

**Apply:** https://hotfix.jobs/jobs/98c20127-42e7-44b8-90a8-23fff60dec09
**Canonical:** https://hotfix.jobs/jobs/98c20127-42e7-44b8-90a8-23fff60dec09