# Staff Engineer, Distributed Storage and HPC & AI Infrastructure

**Company:** [Together AI](https://hotfix.jobs/companies/together-ai)
**Location:** San Francisco, CA
**Role:** DevOps / SRE
**Salary:** $250k – $300k/yr
**Experience:** 8+ years
**Skills:** Wekafs, Ceph, Lustre, Kubernetes, Go, Python, Terraform, Ansible, Helm, Argo CD, Prometheus, Grafana, Rdma, InfiniBand, Nvme-Of
**Posted:** 2026-06-04

> Design and operate multi-petabyte distributed storage systems for large-scale AI training and inference, integrating parallel filesystems and building Kubernetes-native storage platforms.

## Job Description

## Responsibilities
- Design multi-petabyte AI/ML storage systems; integrate WekaFS, Ceph, etc.; lead capacity planning and cost optimization (30-50% savings via tiering, lifecycle policies, right-sizing).
- Design/optimize RDMA, InfiniBand, 400GbE networks; tune for max throughput/min latency; implement NVMe-oF/iSCSI; troubleshoot bottlenecks; optimize TCP/IP for storage.
- Build Kubernetes storage operators/controllers; enable automated provisioning, self-service abstractions, multi-tenant isolation, quotas; create reusable Helm/Terraform patterns.
- Deliver 10-50 GB/s per GPU node; optimize caching (weights/datasets/checkpoints), parallel filesystems, and data paths; troubleshoot with profiling tools; scale to thousands of nodes.
- Build multi-tier caches (local NVMe, distributed, object); optimize data locality and model-weight distribution; implement smart prefetching/eviction.
- Implement monitoring, alerting, SLOs; design DR/backups with runbooks; run chaos engineering; ensure 99.9%+ uptime via proactive/automated remediation.
- Partner with ML/SRE teams; mentor on storage best practices; contribute to open-source; write docs, postmortems, and public learnings.

## Requirements
- 8+ years in storage engineering with 3+ years managing distributed storage at multi-petabyte scale
- Proven track record deploying and operating high-performance storage for GPU/HPC clusters
- Deep Kubernetes and cloud-native storage experience in production environments
- Strong coding skills in Go and Python with demonstrated ability to build production-grade tools
- BS/MS in Computer Science, Engineering, or equivalent practical experience
- History of technical leadership: designing systems that significantly improved performance (>3x), reliability (99.9%+ uptime), or cost efficiency
- Distributed Storage Systems: Deep expertise in WekaFS, Lustre, GPFS, BeeGFS, or similar parallel filesystems at multi-petabyte scale
- Object Storage: Production experience with S3, MinIO, Ceph, or R2 including performance optimization and cost management
- Kubernetes Storage: CSI drivers, StatefulSets, PersistentVolumes, storage operators, and custom controllers
- Storage optimization for GPU workloads, RDMA/InfiniBand networking, parallel filesystem optimization (100+ GB/s aggregate cluster throughput)
- Programming: Go and Python for automation, operators, and tooling
- Infrastructure as Code: Terraform, Ansible, Helm, GitOps (ArgoCD)
- Linux Storage Stack: Advanced knowledge of filesystems (ext4, xfs), LVM, NVMe optimization, RAID configurations
- Observability: Prometheus, Grafana, Thanos architecture and operations

## Nice-to-Haves
- GPU Direct Storage (GDS), NVMe-oF, storage networking (100GbE/400GbE)
- ML/AI storage patterns (model weights, checkpointing, dataset caching)
- Kubernetes operator development (controller-runtime, kubebuilder)
- Storage snapshots, cloning, and thin provisioning
- Backup and disaster recovery (Velero, Restic, cross-region replication)
- Storage encryption (at-rest and in-transit), security and compliance
- Storage benchmarking and profiling tools (fio, iperf3, iostat, blktrace)

## Similar roles

- [Member of Technical Staff](https://hotfix.jobs/jobs/ca5edb5f-58ba-4e0c-a075-25c3fcc61007) - Perplexity - San Francisco, CA - $250k – $405k/yr
- [Staff Site Reliability Engineer](https://hotfix.jobs/jobs/e2420bd8-af00-4aa8-9bcb-34da7d97403d) - Zoox - Foster City, CA - $250k – $300k/yr
- [Staff Engineer, Agent Productivity](https://hotfix.jobs/jobs/b860cf40-a9f4-43ad-961a-fc3aaf976c3d) - Ambience Healthcare - San Francisco, CA - $250k – $300k/yr
- [Member of Technical Staff, Infrastructure](https://hotfix.jobs/jobs/0b3aa2e6-0511-4a76-b1d9-f69a3738db4d) - Envoy - San Francisco, CA - $250k – $280k/yr
- [Staff Site Reliability Engineer](https://hotfix.jobs/jobs/8d2f57a0-c931-4216-805e-4c44749f2506) - Earnin - Mountain View, CA - $252k – $308k/yr

**Apply:** https://hotfix.jobs/jobs/d0fb38e7-169a-4b2d-abb6-6f845d3f381f
**Canonical:** https://hotfix.jobs/jobs/d0fb38e7-169a-4b2d-abb6-6f845d3f381f