HPC Infrastructure Engineer - GPU Clusters

Operates and improves large-scale NVIDIA GPU infrastructure supporting AI model training, spanning provisioning, scheduling, networking, storage, automation, performance tuning, and hardware operations. Requires production Linux or GPU environment experience and strong systems and infrastructure skills.

About the job

Responsibilities

Operate and improve NVIDIA GPU clusters across bare-metal and rented capacity, including provisioning, scheduling, monitoring, upgrades, and capacity planning.
Build automation for node health checks, automated draining and remediation, and burn-in pipelines for new capacity.
Own the infrastructure stack beneath training code, including OS images, NVIDIA drivers, CUDA, container runtimes, NCCL, and InfiniBand/RoCE networking.
Run and tune Slurm or similar job scheduling systems for fair, high-throughput compute access.
Build and maintain high-performance storage for datasets and checkpoints.
Diagnose and resolve performance issues involving stragglers, degraded links, thermal conditions, and unreliable GPUs.
Evaluate rented GPU capacity, benchmark providers, validate performance, and enforce SLAs.
Perform hands-on hardware work, including racking, cabling, diagnostics, and coordination with datacenter staff and vendors.
Maintain secure-by-default clusters through access control, network isolation, and secrets management.

Requirements

Production experience operating large-scale Linux server or GPU environments.
Strong knowledge of the NVIDIA stack, including drivers, CUDA, NCCL, and DCGM, or deep systems experience with the ability to learn hardware stacks quickly.
Experience with bare-metal environments, server hardware, and high-speed networking.
Automation development skills in Python and/or Bash.
Experience with infrastructure-as-code tools such as Ansible or Terraform.
Ability to investigate metrics, logs, and PromQL data to diagnose infrastructure problems.
Willingness to own work end to end and perform datacenter-related tasks when needed.

Nice to Have

Infrastructure experience supporting ML training workloads, including distributed-training failure modes and checkpointing patterns.
Experience evaluating and working with GPU cloud providers.
Experience with parallel filesystems such as WEKA or VAST, or large-scale object storage.
Experience with BMC/IPMI/Redfish automation and PXE provisioning at scale.
Awareness of power and cooling considerations for dense GPU deployments.