Staff Backend Engineer - Application Core Services, Stacks | USA | Remote

Responsibilities

• Design, build, and operate reconciliation systems, including the SSS backend, to track desired stack state, detect and repair drift across stack templates, grafana.com state, Hosted Grafana, and actual customer stack configuration.
• Collaborate across SSS, grafana.com, and deployment configurations to ensure stack lifecycle workflows remain reliable, observable, and resilient.
• Improve operational efficiency by reducing deployment complexity (e.g., aiming for single PR regional SSS deployment) and contributing to the Stack Config Reconciliation project.
• Manage rollout mechanisms for provisioned plugins, dashboards, data sources, Grafana versions, release channels, and stack-level configuration.
• Support new region and cluster rollouts, including the operational paths required to bring stacks online safely in new Grafana Cloud regions.
• Improve incident response and recovery paths for stack misalignment, reconciliation failures, plugin rollout issues, and Hosted Grafana integration failures.
• Partner with Product, Hosted Grafana, Infrastructure, Support, and adjacent AppCore squads on customer-impacting stack lifecycle work.
• Contribute to roadmap planning, technical design, OnCall improvements, and long-term simplification of stack operations.
• Own production behavior of systems, including improving runbooks, dashboards, alerts, reconciliation safety, rollout controls, and recovery procedures.
• Debug across service boundaries and make careful changes in systems affecting customer stacks.

Requirements

• Thrive on building production systems where correctness, scalability, and operational clarity are paramount.
• Comfortable collaborating asynchronously across time zones and taking full ownership of critical systems.
• Lead major projects, coordinate across service boundaries, and help define technical direction.
• Solve challenges related to stateful systems, eventual consistency, and reconciliation loops.
• Transform ambiguous lifecycle requirements into explicit, modular solutions.
• Break down complex systems work into safe, iterative increments.
• Communicate technical tradeoffs to internal stakeholders and adjacent product teams.
• Write efficient, readable, and easy-to-maintain code.
• Comfortable with on-call responsibilities (balanced globally, ~12 daylight hours per day).
• Experience with AI-assisted development practices (e.g., GPT, Claude, Gemini models for prototyping, testing, refactoring).