# Senior Virtualization Validation Engineer

**Company:** [Crusoe](https://hotfix.jobs/companies/crusoe)
**Location:** San Francisco, CA, Sunnyvale, CA
**Role:** DevOps / SRE
**Salary:** $173k – $210k/yr
**Experience:** 5+ years
**Skills:** Qemu, Cloud Hypervisor, Kvm, Nccl, Rccl, CUDA, Rocm, Nvlink, InfiniBand, Roce, Rdma, Sr-Iov, Vfio, Iommu, Pcie Passthrough
**Posted:** 2026-04-20

> Validates large-scale multi-node GPU clusters using QEMU and Cloud Hypervisor, focusing on interconnects like NVLink/InfiniBand, collective communications (NCCL/RCCL), and performance in virtualized AI/HPC environments. Requires 5+ years experience, virtualization expertise, and Linux kernel knowledge.

## Job Description

## What You’ll Be Working On

- **Multi-Node Scaling Validation**: Design and execute large-scale validation tests across multi-node virtualized clusters to ensure linear scaling and stability of GPU workloads.
- **Interconnect & Fabric Testing**: Validate high-speed interconnects—including NVLink, Infinity Fabric, InfiniBand, and RoCE—within virtualized environments to ensure low-latency, high-bandwidth communication.
- **Hypervisor & GPU Virtualization**: Lead the validation of QEMU and Cloud Hypervisor with a focus on PCIe passthrough (VFIO), IOMMU, and direct device assignment for GPUs and high-speed NICs.
- **Collective Communication Benchmarking**: Architect and run comprehensive test suites using nccl-tests and rccl-tests (e.g., AllReduce, AllGather) to verify performance across node boundaries.
- **Network Stack Validation**: Validate SR-IOV and RDMA configurations to ensure that virtualized guests achieve near-bare-metal networking performance for distributed GPU tasks.
- **Automated Cluster Orchestration**: Develop and maintain automation frameworks in Python or Go to dynamically provision, configure, and stress-test multi-node virtualized environments.
- **Performance Bottleneck Analysis**: Perform deep-dive analysis of performance regressions in multi-node communication, identifying root causes across the guest OS, hypervisor, and physical fabric.

## What You’ll Bring to the Team

**Education & Experience**: 5+ YOE demonstrated ability to competently and independently perform responsibilities plus Bachelor’s or Master’s degree in Computer Science, Electrical Engineering, or a related technical field.

**Virtualization Expertise**: Proven experience with QEMU/KVM and Cloud Hypervisor in a production or research environment.

**Distributed GPU Ecosystems**: Deep familiarity with NVIDIA (CUDA/NCCL) and/or AMD (ROCm/RCCL) stacks in a multi-node context.

**Networking Knowledge**: Strong understanding of RDMA, RoCE, and InfiniBand protocols and their implementation in virtualized systems.

**System Internals**: Expert-level knowledge of Linux kernel internals, specifically PCIe topology, VFIO, and memory management (HugePages, IOMMU).

**Automation & Scripting**: Advanced proficiency in Python and/or Bash for automating complex cluster-wide test scenarios.

**Bonus Points**:
- Experience with MNNVL (Multi-Node NVLink) or specialized AI fabric architectures.
- Familiarity with hardware-level debugging tools and performance profilers (e.g., NVIDIA Nsight, AMD Omniperf).
- Knowledge of containerized orchestration for GPUs (e.g., Kubernetes with specialized device plugins).

## Compensation

Compensation will be paid in the range of $172,500 - $210,000. Restricted Stock Units are included in all offers.

## Similar roles

- [Senior Production Engineer, Operational Excellence](https://hotfix.jobs/jobs/17dad742-cf70-4122-940b-da85dad20386) - Crusoe - San Francisco, CA - $172k – $209k/yr
- [Senior Site Reliability Engineer, TCore (FedRamp)](https://hotfix.jobs/jobs/12e08541-0375-44a1-8c3d-76b991365f08) - Okta - San Francisco, CA - $174k – $267k/yr
- [Senior Site Reliability Engineer](https://hotfix.jobs/jobs/8e3a1e08-9bf1-4312-bb71-d6aa18a642fb) - Fivetran - Oakland, CA - $175k – $210k/yr
- [Sr. Software Engineer, Platform Engineering](https://hotfix.jobs/jobs/fafdb79e-697e-4378-aa19-4482d0884fe1) - Chronograph - Remote - $175k – $215k/yr
- [Sr. Site Reliability Engineer](https://hotfix.jobs/jobs/2c837b9b-f230-4e2a-aadb-21d12039b4d2) - Illumio - Sunnyvale, CA - $170k – $196k/yr

**Apply:** https://hotfix.jobs/jobs/d600ecc9-bdcc-4f15-a5e4-c12e28ae1e79
**Canonical:** https://hotfix.jobs/jobs/d600ecc9-bdcc-4f15-a5e4-c12e28ae1e79