# Senior Site Reliability Engineer - Observability

**Company:** [Okta](https://hotfix.jobs/companies/okta)
**Location:** Bellevue, WA, San Francisco, CA
**Role:** DevOps / SRE
**Salary:** $147k – $202k/yr
**Experience:** 5+ years
**Skills:** Splunk, Terraform, Go, Python, Ruby, Spl, Kubernetes, EKS, Linux, OpenTelemetry
**Posted:** 2026-02-26

> Senior SRE specializing in Splunk observability, building scalable platforms with infrastructure as code using Terraform and Go/Python/Ruby. Requires 5+ years Splunk experience and 3+ years SRE in high-availability systems.

## Job Description

## Key Responsibilities
- **Automated Infrastructure**: Design, build, and maintain scalable observability infrastructure using tools like Terraform.
- **Splunk Engineering**: Optimize the collection, processing, and storage of log data to ensure high reliability and low latency of our Splunk services.
- **Incident Response**: Participate in on-call rotations and lead post-incident reviews to drive systemic improvements and "observability-driven development."
- **Automation**: Eliminate "toil" by automating the deployment and scaling of observability agents and collectors.

## Required Skills & Experience (The Essentials)
- **Log Management**: Minimum 5+ years experience scaling and managing Splunk Cloud at scale (1000+ SVCs), including Workload Management (WLM) and HEC optimization.
- **Visualization**: Expertise in creating intuitive, actionable Splunk dashboards that correlate data across multiple sources.
- **SRE Mindset**: Minimum 3+ years of experience in an SRE, DevOps, or Systems Engineering role with a focus on high-availability systems.
- **Programming Proficiency**: Strong coding skills in **SPL**, **Go** for building internal tools and automating workflows.
- **Distributed Systems**: Deep understanding of Linux internals, networking (TCP/IP, DNS, Load Balancing), and container orchestration (Kubernetes/EKS).
- **Problem Solving**: A data-driven approach to debugging complex, cross-service performance bottlenecks.

## Bonus Skills (The "Nice-to-Haves")
- **Telemetry Standards**: Hands-on experience with OpenTelemetry (OTel), Vector, or similar frameworks for instrumenting applications.
- **Charge-back app**: Experience in implementing Splunk charge-back app for usage reporting.
- **Cloud Platforms**: Experience managing observability native tools within AWS or GCP.

## Similar roles

- [Senior Software Engineer, Observability](https://hotfix.jobs/jobs/bd0587f0-116e-41f1-bd5c-6d2027523599) - Okta - Bellevue, WA - $147k – $202k/yr
- [Data Center Systems Engineer (MDC)](https://hotfix.jobs/jobs/740cbbb8-d316-4070-8c61-69bd5f6ba18f) - Crusoe - Denver, CO - $149k – $170k/yr
- [Senior Database Reliability Engineer](https://hotfix.jobs/jobs/35935cb3-7a25-41a1-af42-d17c8ea3b551) - Scribe - Remote - $145k – $230k/yr
- [Sr. Infrastructure Engineer](https://hotfix.jobs/jobs/b067fe70-e3e2-4a67-b2b5-234404fef836) - VGS - Remote - $145k – $185k/yr
- [Senior Systems Software Engineer](https://hotfix.jobs/jobs/d3e5dab9-633a-4a63-ac0f-c834e1209e5a) - Zoo - Remote - $145k – $195k/yr

**Apply:** https://hotfix.jobs/jobs/054d834f-3be0-496b-ba11-ef70b5882bb8
**Canonical:** https://hotfix.jobs/jobs/054d834f-3be0-496b-ba11-ef70b5882bb8