# Software Engineer, Infrastructure

**Company:** [Fal](https://hotfix.jobs/companies/fal)
**Location:** San Francisco, CA
**Role:** DevOps / SRE
**Salary:** $180k – $250k/yr
**Experience:** 5+ years
**Skills:** Python, Linux, Ansible, Terraform, Nvidia Gpus, CUDA, Nvme, Nfs, Selinux, Dcgm, InfiniBand, Lvm, Raid
**Posted:** 2026-02-23

> Build and maintain infrastructure tooling for a large fleet of GPU servers, including provisioning, health monitoring, diagnostics, recovery, storage optimization, and Linux tuning to support AI workloads at scale. Requires 3+ years managing large server fleets, strong Python and deep Linux expertise.

## Job Description

## Key Responsibilities
- Build and maintain Python fleet tracking system managing full lifecycle of servers (contracting, procurement, target use, pricing, availability, health, RMAs).
- Build server management tooling that automates provisioning, health checks, GPU diagnostics, recovery, and alerting.
- Create and maintain metrics, dashboards, and alerting for hardware health across the fleet (GPU errors, disk failures, network issues, thermals).
- Leverage AI extensively to build tools and automate alerting and recovery.
- Implement and enforce OS-level security: hardening baselines, SELinux/AppArmor policies, SSH key management, vulnerability scanning, and compliance automation.
- Manage and optimize distributed and local storage systems (NVMe arrays, NFS, parallel file systems, object storage) supporting model weights, checkpoints, and ephemeral scratch.
- Tune Linux systems for AI workloads: kernel parameters, NUMA topology, CPU pinning, hugepages, I/O schedulers, and GPU driver stack optimization (NVIDIA drivers, CUDA, container runtimes).
- Develop automated error detection and recovery processes.
- Work with partners to solve technical issues.

## Requirements
- 3+ years experience managing bare-metal and cloud-based server fleets at scale (100+ nodes).
- Strong software engineering skills in Python for production tooling.
- Deep Linux systems knowledge: boot process, kernel tuning, networking, storage, systemd, cgroups, namespaces, performance profiling.
- Strong experience with configuration management and infrastructure-as-code (Ansible, Terraform, cloud-init).
- Solid understanding of storage technologies: LVM, RAID, NVMe, NFS, Lustre or GPFS, Linux I/O stack tuning.
- Familiarity with hardware diagnostics and failure modes (GPUs, NVMe, NICs, memory).
- Experience building internal tools or dashboards for infrastructure visibility.
- Excellent communication and ability to drive technical decisions across teams.
- Self-starter who executes quickly, takes ownership, and constantly seeks improvement.

## Nice-to-Haves
- Familiarity with network configuration and diagnostics (VLAN, VXLAN, ECMP, BGP, tcpdump).
- Experience with NVIDIA GPU infrastructure: driver management, health monitoring, DCGM, NVLink/NVSwitch diagnostics, RDMA, InfiniBand/RoCEv2.
- Experience with AMD GPUs.
- Experience with bare metal and VM provisioning (PXE/iPXE, Kickstart, libvirt, Qemu/KVM).
- Experience with compliance frameworks (SOC 2, ISO 27001).

## Compensation
$180,000-250,000 plus equity + benefits.

## Similar roles

- [Platform Engineer](https://hotfix.jobs/jobs/14231c8e-abad-4a41-94e3-d41fb2ede6f5) - Govsignals - New York, NY - $180k – $250k/yr
- [Network Engineer, Design & Engineering](https://hotfix.jobs/jobs/48ff3694-5232-4d66-b43c-a091b96c8902) - Fluidstack - New York, NY - $180k – $300k/yr
- [Developer Productivity Engineer](https://hotfix.jobs/jobs/9a60db26-80d5-4b2a-b613-56ae58d49794) - Hightouch - Remote - $180k – $320k/yr
- [Data Center Network Engineer](https://hotfix.jobs/jobs/473ec43a-a5df-4433-99bd-4053f3e6f6bc) - Baseten - San Francisco, CA - $180k – $360k/yr
- [Infrastructure Engineer (Observability)](https://hotfix.jobs/jobs/098c81d9-0827-4baa-a70b-444cc5b04f37) - Lightning AI - Remote - $180k – $200k/yr

**Apply:** https://hotfix.jobs/jobs/fa972af1-c711-4ae5-9f17-0ea71d2312a7
**Canonical:** https://hotfix.jobs/jobs/fa972af1-c711-4ae5-9f17-0ea71d2312a7