# Infrastructure Engineer (Observability)

**Company:** [Lightning AI](https://hotfix.jobs/companies/lightning-ai)
**Location:** Remote
**Role:** DevOps / SRE
**Salary:** $180k – $200k/yr
**Experience:** 5+ years
**Skills:** Prometheus, Grafana, Kubernetes, Kafka, Otel, Promtail, Python, Go, Elk, Victoriametrics, Nvidia Dcgm, InfiniBand
**Posted:** 2026-05-06

> Builds and operates scalable observability platforms for metrics, logs, traces across GPU, HPC infrastructure. Designs telemetry pipelines, alerting, and multi-tenant systems using Prometheus, Grafana, Kafka; requires 5+ years SRE/infra experience.

## Job Description

## What You’ll Do

### Observability Platform & Productization
- Own and evolve a scalable observability platform spanning metrics, logs, traces, and events
- Drive the productization of observability capabilities for both internal teams and external customers
- Design multi-tenant observability systems with scoped access, **RBAC**, and customer-facing visibility
- Continuously improve observability systems to keep pace with rapid infrastructure buildouts

### Telemetry & Data Pipelines
- Design and operate telemetry pipelines ingesting data from **GPUs**, **CPUs**, networking (**Ethernet** & **InfiniBand**), containers, **APIs**, and **BMC/Redfish**
- Build systems to correlate signals across infrastructure layers to enable faster debugging and root cause analysis
- Implement streaming and real-time data pipelines using tools such as **Kafka**, **OTEL**, **Promtail**, or similar

### Alerting, Reliability & Insights
- Design and implement noise-resistant alerting systems to improve signal quality and reduce operational load
- Create dashboards and alerting for **InfraOps**, Engineering, and **Customer Success** teams
- Build automated insights and enable proactive detection, forecasting, and system health visibility at scale

### Systems & Infrastructure Engineering
- Contribute to broader infrastructure engineering projects beyond observability
- Partner with infrastructure and platform teams to embed observability into core systems and workflows
- Support large-scale, distributed systems across compute, networking, and storage environments

### Cross-Functional Collaboration
- Work closely with customer-facing teams to deliver external observability experiences
- Collaborate with engineering, operations, and support teams to improve system transparency and reliability
- Help define best practices for observability across the organization

## What You’ll Need

### Required Qualifications
- **5+ years** of experience in infrastructure engineering, **SRE**, or observability-focused roles
- Strong experience with monitoring systems such as **Prometheus**, **Grafana**, **ELK**, or **VictoriaMetrics**
- Experience building and operating observability platforms at scale
- Proficiency in **Python**, **Go**, or **bash** for automation and data integration
- Familiarity with containerized environments and **Kubernetes** observability
- Experience with streaming telemetry pipelines (**Kafka**, **OTEL**, **Promtail**, or equivalent)
- Experience with multi-tenant monitoring architectures
- Strong written and verbal communication skills

### Ideal Experience
- Experience with **GPU** observability, particularly **NVIDIA DCGM**
- Experience monitoring large-scale **GPU** or **HPC** clusters
- Familiarity with **InfiniBand** fabric observability
- Experience building customer-facing or productized infrastructure systems
- Experience with correlation engines, **RCA** workflows, or predictive alerting systems
- Broad exposure to infrastructure domains including networking, storage, and provisioning

## Similar roles

- [Platform Engineer](https://hotfix.jobs/jobs/14231c8e-abad-4a41-94e3-d41fb2ede6f5) - Govsignals - New York, NY - $180k – $250k/yr
- [Network Engineer, Design & Engineering](https://hotfix.jobs/jobs/48ff3694-5232-4d66-b43c-a091b96c8902) - Fluidstack - New York, NY - $180k – $300k/yr
- [Developer Productivity Engineer](https://hotfix.jobs/jobs/9a60db26-80d5-4b2a-b613-56ae58d49794) - Hightouch - Remote - $180k – $320k/yr
- [Data Center Network Engineer](https://hotfix.jobs/jobs/473ec43a-a5df-4433-99bd-4053f3e6f6bc) - Baseten - San Francisco, CA - $180k – $360k/yr
- [Infrastructure Engineer (GPU & Compute)](https://hotfix.jobs/jobs/2dd5ac53-2601-447b-af6d-e643a08de3ad) - Lightning AI - Remote - $180k – $200k/yr

**Apply:** https://hotfix.jobs/jobs/098c81d9-0827-4baa-a70b-444cc5b04f37
**Canonical:** https://hotfix.jobs/jobs/098c81d9-0827-4baa-a70b-444cc5b04f37