Skip to content
OktaOktaBellevue, WA

Senior Site Reliability Engineer - Observability

Senior SRE specializing in Splunk observability, building scalable platforms with infrastructure as code using Terraform and Go/Python/Ruby. Requires 5+ years Splunk experience and 3+ years SRE in high-availability systems.

147k – 202k/yr
Hybrid5+ YOEDevOps / SRE

About the role

Key Responsibilities

  • Automated Infrastructure: Design, build, and maintain scalable observability infrastructure using tools like Terraform.
  • Splunk Engineering: Optimize the collection, processing, and storage of log data to ensure high reliability and low latency of our Splunk services.
  • Incident Response: Participate in on-call rotations and lead post-incident reviews to drive systemic improvements and "observability-driven development."
  • Automation: Eliminate "toil" by automating the deployment and scaling of observability agents and collectors.

Required Skills & Experience (The Essentials)

  • Log Management: Minimum 5+ years experience scaling and managing Splunk Cloud at scale (1000+ SVCs), including Workload Management (WLM) and HEC optimization.
  • Visualization: Expertise in creating intuitive, actionable Splunk dashboards that correlate data across multiple sources.
  • SRE Mindset: Minimum 3+ years of experience in an SRE, DevOps, or Systems Engineering role with a focus on high-availability systems.
  • Programming Proficiency: Strong coding skills in SPL, Go for building internal tools and automating workflows.
  • Distributed Systems: Deep understanding of Linux internals, networking (TCP/IP, DNS, Load Balancing), and container orchestration (Kubernetes/EKS).
  • Problem Solving: A data-driven approach to debugging complex, cross-service performance bottlenecks.

Bonus Skills (The "Nice-to-Haves")

  • Telemetry Standards: Hands-on experience with OpenTelemetry (OTel), Vector, or similar frameworks for instrumenting applications.
  • Charge-back app: Experience in implementing Splunk charge-back app for usage reporting.
  • Cloud Platforms: Experience managing observability native tools within AWS or GCP.

Skills

SplunkTerraformGoPythonRubySplKubernetesEKSLinuxOpenTelemetry

Similar roles

DevOps / SRE jobs
Okta

Senior Software Engineer, Observability

OktaBellevue, WA +3

Senior engineer on the Auth0 Platform Observability team responsible for designing, building, and maintaining scalable observability infrastructure (metrics, logs, traces) using Datadog, Terraform, and OpenTelemetry.

147k – 202k/yr
Hybrid5+ YOEDevOps / SRE
Crusoe

Data Center Systems Engineer (MDC)

CrusoeDenver, CO +1

Leads integration of electrical, thermal, mechanical, and networking systems in modular data centers for high-density AI compute. Ensures compatibility with power sources, optimizes thermal management, and supports transition to liquid cooling with 6+ years systems engineering experience.

149k – 170k/yr
On-site6+ YOEDevOps / SRE
Scribe

Senior Database Reliability Engineer

ScribeSan Francisco, CA

Senior IC role owning reliability, performance, and scalability of PostgreSQL (Aurora), OpenSearch, Redis, and CDC pipeline to Snowflake. Sets standards for ORM usage, migration safety, and observability at scale.

145k – 230k/yr
Remote5+ YOEDevOps / SRE
VGS

Sr. Infrastructure Engineer

VGSUnited States

As a Senior Infrastructure Engineer, you will be responsible for architecting and maintaining scalable, reliable cloud infrastructure, leading incident management, and improving operational processes. This role requires strong proficiency in AWS, infrastructure-as-code, and experience with monitoring and observability tools.

145k – 185k/yr
Remote5+ YOEDevOps / SRE
Zoo

Senior Systems Software Engineer

ZooLos Angeles, CA

Builds and maintains core backend systems, infrastructure, and automation for platform reliability and scalability. Owns troubleshooting across stack, integrations with external services, and observability. Requires strong Rust, systems engineering, and distributed systems experience.

145k – 195k/yr
RemoteDevOps / SRE