# Staff Site Reliability Engineer - Observability

**Company:** [Okta](https://hotfix.jobs/companies/okta)
**Location:** Bellevue, WA, Chicago, IL, New York, NY, San Francisco, CA, Washington, DC
**Role:** DevOps / SRE
**Salary:** $194k – $267k/yr
**Experience:** 5+ years
**Skills:** Terraform, GKE, Splunk, Grafana, Python, Go, Kubernetes, Linux, TCP/IP, DNS, Load Balancing, OpenTelemetry
**Posted:** 2026-06-17

> Staff SRE focused on building and scaling a comprehensive observability platform on GCP using Terraform, Splunk, and Grafana. Requires 5+ years GCP observability experience and strong coding skills in Python or Go.

## Job Description

## Key Responsibilities
- Design, build, and maintain scalable observability infrastructure using tools like Terraform
- Optimize the collection, processing, and storage of Observability data to ensure high reliability and low latency of Splunk and Grafana services on GCP
- Participate in on-call rotations and lead post-incident reviews to drive systemic improvements and "observability-driven development"
- Automate the deployment and scaling of observability agents and collectors to eliminate toil

## Requirements
- Minimum 5+ years scaling and managing observability in Google Cloud Platform (GKE)
- Expertise in creating intuitive, actionable Splunk or Grafana dashboards that correlate data across multiple sources
- Minimum 3+ years of experience in an SRE, DevOps, or Systems Engineering role with a focus on high-availability systems
- Strong coding skills in Python or Go for building internal tools and automating workflows
- Deep understanding of Linux internals, networking (TCP/IP, DNS, Load Balancing), and container orchestration (Kubernetes/GKE)
- Data-driven approach to debugging complex, cross-service performance bottlenecks

## Nice-to-Haves
- Hands-on experience with OpenTelemetry (OTel), Vector, or similar frameworks
- Experience migrating Splunk to Grafana Loki
- Experience managing observability native tools within AWS

This position requires U.S. Person status.

## Similar roles

- [Staff Engineer](https://hotfix.jobs/jobs/c04f6e72-1a89-4d07-af71-b851172954d7) - Illumio - Sunnyvale, CA - $194k – $233k/yr
- [Staff Platform Engineer - Infra + DevOps](https://hotfix.jobs/jobs/9af9ab09-1316-4f67-b5c3-069f17cadfda) - Honor - Remote - $194k – $220k/yr
- [Staff Software Engineer, Kubernetes Cloud](https://hotfix.jobs/jobs/ce81c371-0f7c-40f8-8e07-83780b83707b) - Illumio - Sunnyvale, CA - $194k – $233k/yr
- [Staff/Senior Software Engineer, Offboard Infrastructure](https://hotfix.jobs/jobs/9fab1e3f-5567-4470-8d80-d04bb25e210f) - Nuro - Mountain View, CA - $194k – $352k/yr
- [Staff Network Engineer, Operations](https://hotfix.jobs/jobs/19d77267-e1da-45b5-9ed9-fce44a90d734) - Crusoe - San Francisco, CA - $195k – $235k/yr

**Apply:** https://hotfix.jobs/jobs/9b77ecd4-1246-4f70-a6af-3262fc1aaca6
**Canonical:** https://hotfix.jobs/jobs/9b77ecd4-1246-4f70-a6af-3262fc1aaca6