Skip to content
FalFalSan Francisco, CA

Software Engineer, Infrastructure

Build and maintain infrastructure tooling for a large fleet of GPU servers, including provisioning, health monitoring, diagnostics, recovery, storage optimization, and Linux tuning to support AI workloads at scale. Requires 3+ years managing large server fleets, strong Python and deep Linux expertise.

180k – 250k/yr
Hybrid5+ YOEDevOps / SRE

About the role

Key Responsibilities

  • Build and maintain Python fleet tracking system managing full lifecycle of servers (contracting, procurement, target use, pricing, availability, health, RMAs).
  • Build server management tooling that automates provisioning, health checks, GPU diagnostics, recovery, and alerting.
  • Create and maintain metrics, dashboards, and alerting for hardware health across the fleet (GPU errors, disk failures, network issues, thermals).
  • Leverage AI extensively to build tools and automate alerting and recovery.
  • Implement and enforce OS-level security: hardening baselines, SELinux/AppArmor policies, SSH key management, vulnerability scanning, and compliance automation.
  • Manage and optimize distributed and local storage systems (NVMe arrays, NFS, parallel file systems, object storage) supporting model weights, checkpoints, and ephemeral scratch.
  • Tune Linux systems for AI workloads: kernel parameters, NUMA topology, CPU pinning, hugepages, I/O schedulers, and GPU driver stack optimization (NVIDIA drivers, CUDA, container runtimes).
  • Develop automated error detection and recovery processes.
  • Work with partners to solve technical issues.

Requirements

  • 3+ years experience managing bare-metal and cloud-based server fleets at scale (100+ nodes).
  • Strong software engineering skills in Python for production tooling.
  • Deep Linux systems knowledge: boot process, kernel tuning, networking, storage, systemd, cgroups, namespaces, performance profiling.
  • Strong experience with configuration management and infrastructure-as-code (Ansible, Terraform, cloud-init).
  • Solid understanding of storage technologies: LVM, RAID, NVMe, NFS, Lustre or GPFS, Linux I/O stack tuning.
  • Familiarity with hardware diagnostics and failure modes (GPUs, NVMe, NICs, memory).
  • Experience building internal tools or dashboards for infrastructure visibility.
  • Excellent communication and ability to drive technical decisions across teams.
  • Self-starter who executes quickly, takes ownership, and constantly seeks improvement.

Nice-to-Haves

  • Familiarity with network configuration and diagnostics (VLAN, VXLAN, ECMP, BGP, tcpdump).
  • Experience with NVIDIA GPU infrastructure: driver management, health monitoring, DCGM, NVLink/NVSwitch diagnostics, RDMA, InfiniBand/RoCEv2.
  • Experience with AMD GPUs.
  • Experience with bare metal and VM provisioning (PXE/iPXE, Kickstart, libvirt, Qemu/KVM).
  • Experience with compliance frameworks (SOC 2, ISO 27001).

Compensation

$180,000-250,000 plus equity + benefits.

Skills

PythonLinuxAnsibleTerraformNvidia GpusCUDANvmeNfsSelinuxDcgmInfiniBandLvmRaid

Similar roles

DevOps / SRE jobs
Govsignals

Platform Engineer

GovsignalsNew York, NY

Platform Engineer building and scaling data ingestion, storage, search, and deployment infrastructure across commercial, FedRAMP High, and IL5 environments. Requires 5+ years in backend/platform/infra roles with deep experience in Postgres, Kubernetes, and TypeScript/Python.

180k – 250k/yr
Hybrid5+ YOEDevOps / SRE
Fluidstack

Network Engineer, Design & Engineering

FluidstackNew York, NY +4

Design end-to-end datacenter network architectures for AI training and inference workloads. Own topology selection, fabric design, physical infrastructure integration, and produce deployable HLDs/LLDs across multiple GPU platforms and customer requirements.

180k – 300k/yr
On-site5+ YOEDevOps / SRE
Hightouch

Developer Productivity Engineer

HightouchUnited States

As a Senior Developer Productivity Engineer, you will own the build, test, and deployment processes for a 50+ person engineering team. You will improve monorepo productivity, drive excellence in testing, and support multi-cloud/multi-region infrastructure to enable fast and safe shipping.

180k – 320k/yr
Remote5+ YOEDevOps / SRE
Baseten

Data Center Network Engineer

BasetenSan Francisco, CA +1

Design and own high-performance data center network infrastructure for GPU clusters, including fabric architecture, cabling, and performance validation. Requires deep experience with InfiniBand, RDMA, or high-performance Ethernet at a senior level.

180k – 360k/yr
Hybrid5+ YOEDevOps / SRE
Lightning AI

Infrastructure Engineer (Observability)

Lightning AINew York, NY +2

Builds and operates scalable observability platforms for metrics, logs, traces across GPU, HPC infrastructure. Designs telemetry pipelines, alerting, and multi-tenant systems using Prometheus, Grafana, Kafka; requires 5+ years SRE/infra experience.

180k – 200k/yr
Remote5+ YOEDevOps / SRE