# Senior/Staff Kubernetes Infrastructure Engineer

**Company:** [Fal](https://hotfix.jobs/companies/fal)
**Location:** Remote
**Role:** DevOps / SRE
**Salary:** $180k – $250k/yr
**Experience:** 5+ years
**Skills:** Kubernetes, Linux, slurm, nvidia gpus, gpu operator, kvm/qemu, libvirt, vfio, cilium, calico, Ansible, Python, Go, ceph, BGP
**Posted:** 2026-08-14

> Build and operate high-performance customer compute environments spanning bare metal, Kubernetes, Slurm, GPUs, networking, storage, and observability. The role requires 5+ years of production Linux infrastructure experience and strong expertise in bare-metal Kubernetes, NVIDIA GPUs, virtualization, and networking.

## Job Description

## Responsibilities
- Design, automate, validate, and deliver the complete lifecycle of customer compute environments, from provisioning through upgrades, recovery, and decommissioning.
- Use AI to automate and accelerate infrastructure delivery and operations.
- Provision dedicated Kubernetes and Slurm clusters tailored to customer workloads.
- Build and maintain Linux images and automated OS-provisioning workflows.
- Operate the NVIDIA GPU stack, including drivers, GPU Operator, NVIDIA Container Toolkit, device plugins, MIG, and GPU monitoring.
- Design Kubernetes and data-center networking using Cilium/Calico, MetalLB, VLAN, VXLAN, BGP, and ECMP.
- Configure distributed and shared storage for high-performance workloads.
- Build monitoring, alerting, diagnostics, and automated recovery for customer environments.
- Develop reusable tooling, standards, documentation, and runbooks.
- Collaborate with customers and internal teams to translate workload requirements into infrastructure designs.

## Requirements
- 5+ years of experience building and operating production Linux infrastructure.
- Strong production experience with Kubernetes on bare metal, including bootstrapping, upgrades, highly available control planes, etcd, containerd, CNI, CSI, ingress, load balancing, observability, security, and troubleshooting.
- Experience with Linux virtualization using KVM/QEMU, libvirt, and VFIO device passthrough.
- Experience operating NVIDIA GPUs on Linux and Kubernetes, including drivers, container runtimes, device plugins, GPU Operator, and GPU telemetry.
- Strong networking fundamentals, including TCP/IP, L2/L3, VLANs, routing, and packet-level troubleshooting with tcpdump and Wireshark.
- Practical scripting experience.
- Experience with configuration-management tools such as Ansible.
- Ability to diagnose complex, cross-layer infrastructure issues.
- Strong communication skills and the ability to drive technical decisions across teams.
- Track record of moving quickly, taking ownership, and continuously improving systems.

## Nice to Have
- Production Slurm experience.
- High-performance networking: NVLink/NVSwitch, InfiniBand, RoCEv2, GPUDirect RDMA, NCCL, and IMEX.
- Hugepages, NUMA, and CPU pinning.
- SR-IOV and DPDK.
- Distributed storage: Ceph, Lustre, and Weka.
- KubeVirt and OpenStack.
- IPsec, WireGuard, and Tailscale.
- VXLAN, BGP, and ECMP.
- Bare-metal management: BMC, IPMI, Redfish, PXE/iPXE, Kickstart, and cloud-init.
- Network automation with NetBox, Nautobot, and Nornir.
- AI training, inference, or distributed GPU workload infrastructure.
- Python or Go proficiency.

## Similar roles

- [Staff Site Reliability Engineer](https://hotfix.jobs/jobs/87068331-357d-4049-9cef-b6b8edc93f79) - Attentive - Remote - $180k – $240k/yr
- [Sr. Staff Platform/Data Reliability Engineer, Databricks](https://hotfix.jobs/jobs/b1ca8d0a-2503-4891-8213-a0c0ca8c01ad) - Shield AI - Remote - $180k – $270k/yr
- [Staff Engineer, AI Productivity](https://hotfix.jobs/jobs/8dcfdcba-4e0d-400e-a95e-d01acbf151e3) - Hightouch - Remote - $180k – $400k/yr
- [Staff Software Engineer, AI Developer Tools](https://hotfix.jobs/jobs/4ad0f960-2ad7-4648-829d-c4aff746157f) - Gusto - Denver, CO - $180k – $245k/yr
- [Staff Infrastructure Engineer](https://hotfix.jobs/jobs/31b641c0-5800-45dd-bdac-4948a91afb52) - Onebrief - Remote - $180k – $235k/yr

**Apply:** https://hotfix.jobs/jobs/5276ef82-8104-4269-8e3f-7f0e02d35c2d
**Canonical:** https://hotfix.jobs/jobs/5276ef82-8104-4269-8e3f-7f0e02d35c2d