Staff Software Engineer, GPU Inference

Responsibilities

• Design, build, deploy, and maintain the complete GPU prefill path across API services, model-serving workers, vLLM, PyTorch, ROCm, GPU nodes, networking, and rack-scale infrastructure.
• Establish deployment, upgrade, rollback, health-checking, capacity-management, and failure-recovery practices for an AMD GPU fleet.
• Build automation to make driver, firmware, runtime, model, and container compatibility explicit and reproducible.
• Define service-level indicators and objectives for GPU-backed inference; improve fault isolation, graceful degradation, automated recovery, incident response, and post-incident remediation.
• Optimize time to first token, request throughput, tokens per second per GPU, tail latency, GPU utilization, memory efficiency, and rack-level capacity.
• Tune scheduling, continuous batching, prefix caching, KV-cache management, tensor and expert parallelism, request admission, quantization, graph execution, and distributed communication.
• Diagnose failures and performance regressions across application code, vLLM, PyTorch, ROCm/HIP, collective communication libraries, kernels, drivers, firmware, networking, and hardware.
• Build validation and regression infrastructure for model quality, numerical accuracy, precision changes, quantization, determinism, and software/hardware compatibility.
• Develop benchmarks, workload-replay tools, profiling automation, release qualification, dashboards, and regression gates.

Requirements

• 8+ years of software engineering experience, including substantial individual-contributor ownership of complex production systems.
• Experience building, operating, or optimizing production inference systems for large language models, multimodal models, or similarly demanding GPU workloads.
• Strong programming ability in C++ and Python, including multithreading, concurrency, memory management, and performance-sensitive software.
• Hands-on experience with vLLM, SGLang, TensorRT-LLM, Triton Inference Server, or an equivalent model-serving system.
• Strong understanding of GPU execution and performance, including asynchronous execution, memory movement, synchronization, kernel launches, communication overhead, and profiling.
• Experience debugging distributed systems across multiple layers.
• Experience with Linux, containers, Kubernetes or comparable orchestration systems, observability, CI/CD, and latency-sensitive production services.
• Ability to design rigorous benchmarks, interpret noisy performance results, identify bottlenecks, and translate findings into production improvements.
• Strong communication and technical leadership skills.
• Bachelor’s degree in Computer Science, Computer Engineering, Electrical Engineering, or a related discipline, or equivalent practical experience.

Nice-to-haves

• AMD Instinct accelerators and the ROCm ecosystem, including HIP, RCCL, rocprofiler, AMD SMI, AITER, hipBLASLt, and Composable Kernel.
• Deep CUDA experience.
• Contributions to vLLM, SGLang, PyTorch, Triton, TensorRT-LLM, or another open-source ML systems project.
• Prefill-heavy or disaggregated prefill/decode inference architectures.
• KV-cache transfer, prefix caching, continuous batching, chunked prefill, request scheduling, and memory-aware admission control.
• Multi-GPU and multi-node inference, tensor parallelism, pipeline parallelism, expert parallelism, RDMA, collective communication, and failure handling.
• Mixture-of-Experts or multimodal models.
• GPU kernel optimization, operator fusion, graph capture, attention kernels, GEMM tuning, and communication/computation overlap.
• Reduced-precision inference and quantization formats including BF16, FP8, FP4, INT8, and INT4.
• Numerical-comparison, determinism, model-validation, or performance-regression test systems.
• Collaboration with accelerator vendors, framework maintainers, or open-source communities.