# Staff Software Engineer, Observability

**Company:** [Temporal](https://hotfix.jobs/companies/temporal)
**Location:** Remote
**Role:** Backend Engineering
**Salary:** $170k – $278k/yr
**Experience:** 7+ years
**Skills:** Go, Kubernetes, SQL, AWS, GCP, ClickHouse, Prometheus, Grafana, Loki, Thanos, Temporal, Distributed Systems, Data Streaming, Networking, Operating Systems
**Posted:** 2026-04-29

> Leads the design, development, deployment, and operation of large-scale observability and distributed systems serving internal developers and customers. Requires staff-level expertise in Kubernetes, programming, query development, cloud infrastructure, and production reliability.

## Job Description

## Responsibilities
- Lead the end-to-end software development lifecycle, including requirements gathering, design, implementation, deployment, operationalization, support, and maintenance.
- Lead feature design and reviews, incorporate stakeholder feedback, and drive consensus.
- Document design decisions and operational knowledge for successful deployment and management.
- Ensure unit, integration, performance, and production-readiness testing.
- Design and build horizontally scalable, multi-component distributed systems.
- Investigate issues methodically, identify root causes, and assess performance and reliability tradeoffs at scale.
- Participate in the on-call rotation.
- Develop expert knowledge of the assigned domain and the Temporal ecosystem.
- Understand internal developer and external customer needs to guide product development and feature design.
- Mentor teammates, participate in design reviews, and contribute to reliable-system design.

## Requirements
- Demonstrated experience developing horizontally scalable, resilient, high-performance distributed systems in production.
- Experience designing, implementing, deploying, and supporting large-scale, geographically distributed observability systems, high-throughput data streaming or processing pipelines, or similar systems.
- Expert proficiency in one or more high-level programming languages, preferably Go.
- Expert-level Kubernetes skills.
- Expert-level query development skills, preferably SQL.
- Hands-on experience with a cloud provider, preferably AWS or Google Cloud.
- Thorough understanding of computer architecture, operating systems, and networking.
- Familiarity with monitoring, instrumentation, and infrastructure configuration best practices.
- Strong collaboration, communication, self-direction, and user-focused problem-solving skills.

## Similar jobs

- [Staff Scanning Engineer](https://hotfix.jobs/jobs/2275cea4-b117-4872-8690-7ec9a2509ade) - Censys - Remote - $172k – $216k/yr
- [Staff Backend Engineer - Grafana App Platform| US| Remote](https://hotfix.jobs/jobs/3fb8d520-afe7-4dd7-ade5-522491c26c83) - Grafana Labs - Remote - $175k – $210k/yr
- [Staff Ground Software Engineer](https://hotfix.jobs/jobs/55cfb3aa-6810-409a-8df0-8e9ada079432) - Turion Space - Irvine, CA - $175k – $225k/yr
- [Staff Software Engineer, Metadata](https://hotfix.jobs/jobs/16e23b4d-e57c-4aa5-a453-30ec7d50efaa) - Fivetran - Remote - $162k – $199k/yr
- [Staff Software Engineer, Metadata](https://hotfix.jobs/jobs/3aba1f31-fbd1-4d79-97b5-de7abf0aa37c) - Fivetran - Remote - $162k – $199k/yr

**Apply:** https://hotfix.jobs/jobs/f427bca2-f1d5-4507-804f-7f4e095c0d94
**Canonical:** https://hotfix.jobs/jobs/f427bca2-f1d5-4507-804f-7f4e095c0d94