# Software Engineer, Infrastructure

**Company:** [Fal](https://hotfix.jobs/companies/fal)
**Location:** Remote
**Role:** DevOps / SRE
**Experience:** 3+ years
**Skills:** Python, Linux, Ansible, Terraform, Nvidia, CUDA, Selinux, Apparmor, Nvme, Nfs, Lustre, Gpfs, Dcgm, InfiniBand, Rocev2
**Posted:** 2026-03-13

> Build and maintain infrastructure for a large fleet of GPU servers, including provisioning, health monitoring, diagnostics, recovery, storage optimization, and Linux tuning for AI workloads. Requires 3+ years managing large-scale bare-metal/cloud fleets, strong Python and deep Linux expertise.

## Job Description

## Key Responsibilities
- Build and maintain Python fleet tracking system that manages the full lifecycle of servers including contracting and procurement, target use, pricing, availability, health, RMAs, etc.
- Build server management tooling that automates provisioning, health checks, GPU diagnostics, recovery and alerting.
- Create and maintain metrics, dashboards, and alerting for hardware health across the fleet (GPU errors, disk failures, network issues, thermals).
- Leverage AI to an extreme level to build tools and automate alerting and recovery.
- Implement and enforce OS-level security: hardening baselines, SELinux/AppArmor policies, SSH key management, vulnerability scanning, and compliance automation.
- Manage and optimize distributed and local storage systems supporting model weights, checkpoints, and ephemeral scratch: NVMe arrays, NFS, parallel file systems, and object storage.
- Tune Linux systems for AI workloads: kernel parameters, NUMA topology, CPU pinning, hugepages, I/O schedulers, and GPU driver stack optimization (NVIDIA drivers, CUDA, container runtimes).
- Develop a suite of automated error detection and recovery processes.
- Work with partners to solve technical issues.

## Requirements
- 3+ years experience managing bare-metal and cloud based server fleets at scale (100+ nodes).
- Strong software engineering skills in Python; you write production tooling, not scripts.
- Deep Linux systems knowledge: boot process, kernel tuning, networking, storage, systemd, cgroups, namespaces, performance profiling.
- Strong experience with configuration management and infrastructure-as-code: Ansible, Terraform, cloud-init.
- Solid understanding of storage technologies: LVM, RAID, NVMe, NFS, Lustre or GPFS, and Linux I/O stack tuning.
- Familiarity with hardware diagnostics and failure modes (GPUs, NVMe, NICs, memory).
- Experience building internal tools or dashboards for infrastructure visibility.
- Excellent communication and ability to drive technical decisions across teams.
- Self-starter who executes quickly, takes ownership, and constantly seeks improvement.

## Nice-to-Haves
- Familiarity with network configuration and diagnostics (VLAN, VXLAN, ECMP, BGP, tcpdump).
- Experience with NVIDIA GPU infrastructure: driver management, health monitoring, DCGM, NVLink/NVSwitch diagnostics, RDMA, InfiniBand/RoCEv2.
- Experience with AMD GPUs.
- Experience with bare metal and VM provisioning (PXE/iPXE, Kickstart, libvirt, Qemu/KVM).
- Experience with compliance frameworks relevant to cloud providers (SOC 2, ISO 27001).

## Similar roles

- [Software Engineer, Platform & Infrastructure](https://hotfix.jobs/jobs/2a4d6818-4e00-4a08-8339-aba841ed1ada) - Forus - New York, NY
- [Software Engineer, Developer Productivity](https://hotfix.jobs/jobs/3d3eb78d-cb73-4b62-bef0-6c84085fdf5a) - Forus - New York, NY
- [Site Reliability Engineer](https://hotfix.jobs/jobs/8def48bc-b809-4397-bd55-28b9747c1f82) - Runloop - San Francisco, CA
- [Build & Release Engineer](https://hotfix.jobs/jobs/c41c87c5-6007-46e9-8648-635789c39b4a) - Applied Intuition - Sunnyvale, CA - $118k – $200k/yr
- [Software Engineer, Async Platform](https://hotfix.jobs/jobs/f1db6fa3-5b80-4939-86b0-39e6f0db0694) - Lyft - Seattle, WA - $118k – $147k/yr

**Apply:** https://hotfix.jobs/jobs/1ce3b9bd-2333-4bfc-bb32-f4a97ff23214
**Canonical:** https://hotfix.jobs/jobs/1ce3b9bd-2333-4bfc-bb32-f4a97ff23214