Senior Software Engineer, Observability

Responsibilities

• Design and implement a scalable observability platform (metrics, logs, traces) using tools like Prometheus, Grafana, ClickHouse, ClickStack, and OpenTelemetry, including telemetry data pipelines and log aggregation workflows.
• Develop automated monitoring, alerting, and anomaly detection systems, including SLIs/SLOs, runbooks, and predictive analytics for critical services.
• Build and deploy custom observability tools and infrastructure-as-code using Go, Python, Terraform, Ansible, and Helm.
• Collaborate with engineering teams to enhance distributed tracing and application monitoring, and lead incident response with post-mortem analysis.
• Define observability best practices.

Requirements

• Expertise in observability platforms (Prometheus, Grafana, ClickStack, OpenTelemetry) and cloud-native monitoring services (AWS, GCP, Azure).
• Strong programming skills in Go, Python, or similar languages, with proficiency in infrastructure-as-code tools (Terraform, Ansible, Helm).
• Experience designing, operating, and scaling large-scale distributed systems and pipelines for high-volume data ingestion and real-time querying.
• Deep understanding of containerization (Docker) and orchestration (Kubernetes).
• Knowledge of microservices architecture, service mesh technologies, CI/CD pipelines, and GitOps workflows.
• Expertise in managing databases (PostgreSQL, MongoDB, Redis) and time-series databases with high-cardinality data.

Preferred

• Experience monitoring AI/ML infrastructure, GPU clusters, and custom metrics for model performance and training pipelines.
• Background in high-frequency, low-latency systems monitoring, chaos engineering, and reliability testing.
• Contributions to open-source observability projects.
• Familiarity with security monitoring and compliance frameworks.

Compensation

• US base salary range for this full-time position is: $200,000 - $280,000 + equity + benefits.