# Software Engineer

**Company:** [Baseten](https://hotfix.jobs/companies/baseten)
**Location:** San Francisco, CA
**Role:** DevOps / SRE
**Salary:** $200k – $380k/yr
**Experience:** 7+ years
**Skills:** Distributed Systems, Telemetry Pipelines, Streaming Systems, High-Cardinality Data, Anomaly Detection, Root Cause Analysis, Networking, Rdma, Gpudirect, Kubernetes, InfiniBand, Roce
**Posted:** 2026-07-07

> Lead Software Engineer building real-time observability and root-cause analysis for Baseten's large-scale GPU fabrics and high-performance networks. Requires strong distributed systems and infrastructure experience to correlate telemetry from switches, NICs, GPUs, and inference services.

## Job Description

## Responsibilities
- Own Baseten’s GPU fabric observability and root-cause analysis architecture.
- Build telemetry pipelines across switches, NICs, hosts, GPUs, Kubernetes, and inference services.
- Model topology, flow paths, service ownership, and failure domains.
- Separate true fabric faults from host, NIC, GPU, kernel, driver, RDMA, scheduler, and workload failures.
- Create clear operator workflows for triage, remediation, and post-incident learning.

## Example Initiatives
- Real-time telemetry engine — Build the ingestion, reduction, storage, and query path for high-cardinality fabric, host, GPU, and workload telemetry.
- Service-aware fabric diagnosis — Build collectors, probes, and topology-aware correlation to detect latency, drops, stalls, congestion, bad paths, and degradation.
- Software-aware RDMA diagnosis — Tie network behavior to RDMA operations, GPUDirect paths, KV cache transfers, prefill/decode disaggregation, and request latency.

## Requirements
- Staff-level or senior staff-level experience building production infrastructure software.
- Strong distributed systems background, especially streaming systems, telemetry pipelines, diagnostics, or control-plane software.
- Experience building systems that process high-volume, high-cardinality, noisy operational data.
- Understanding of networking fundamentals and high-performance networks.
- Ability to work with low-level infrastructure signals and build practical correlation, anomaly detection, or root-cause analysis systems.

## Nice-to-Haves
- Experience with RDMA data paths, GPUDirect transfers, prefill/decode disaggregation, KV cache movement, request routing, and workload backpressure.
- Familiarity with RoCE, InfiniBand, ECN/DCQCN, RNIC, GPU drivers, and related technologies.

## Benefits
- Competitive compensation, including meaningful equity.
- 100% coverage of medical, dental, and vision insurance for employee and dependents.
- Flexible PTO policy including company wide Winter Break (our offices are closed from Christmas Eve to New Year's Day!).
- Paid parental leave.
- Fertility and family-building stipend through Carrot.
- Company-facilitated 401(k).
- Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities.

## Similar roles

- [Senior Software Engineer, Cloud Infrastructure](https://hotfix.jobs/jobs/78aefb2b-cbc3-4660-bce1-7ac2fd28f1c0) - Decagon - San Francisco, CA - $200k – $400k/yr
- [Senior Software Engineer, Observability](https://hotfix.jobs/jobs/b857fbeb-7984-45df-bd84-c85c53858f73) - Together AI - San Francisco, CA - $200k – $280k/yr
- [Senior Software Engineer, Accelerated Delivery](https://hotfix.jobs/jobs/d6170467-5d52-4dcf-bf3d-e9475a94e8f6) - Snowflake - Menlo Park, CA - $200k – $288k/yr
- [Senior Software Engineer - Snowpark Container Service](https://hotfix.jobs/jobs/47626f55-409c-4469-9cea-906cfd5683e3) - Snowflake - Bellevue, WA - $200k – $288k/yr
- [Lead Site Reliability Engineer](https://hotfix.jobs/jobs/212a6b4f-73ba-418c-b30c-cda198e4b506) - Stuut - San Francisco, CA - $200k – $275k/yr

**Apply:** https://hotfix.jobs/jobs/e2716385-59d1-4fb7-9c9b-ebd5ec347f2f
**Canonical:** https://hotfix.jobs/jobs/e2716385-59d1-4fb7-9c9b-ebd5ec347f2f