Skip to content
xAIxAIPalo Alto, CA

Member of Technical Staff, Inference

Build and optimize the high-performance inference platform serving Grok at massive scale. Design distributed serving infrastructure, low-level GPU optimizations, quantization, speculative decoding, and CI/CD for production reliability and low latency.

180k – 440k
On-site5+ YOEML Engineering

About the role

Responsibilities

  • Architect and implement scalable distributed infrastructure for model serving (load balancing, auto-scaling, batch scheduling, global KV cache).
  • Optimize latency and throughput of model inference under real production workloads.
  • Build reliable, high-concurrency serving systems that serve billions of users with 100% uptime, 0% error rate, and excellent tail latency.
  • Benchmark, fine-tune, and accelerate inference engines (including low-level GPU kernel work and code generation).
  • Develop custom tools to trace, replay, and fix issues across the full stack — from orchestration down to GPU kernels.
  • Create robust CI/CD infrastructure for seamless endpoint deployment, image publishing, and inference engine updates.
  • Accelerate research on scaling test-time compute, RL rollout, and model-hardware co-design for next-generation systems.

Requirements

  • Deep low-level systems programming (C/C++ or Rust).
  • Experience with large-scale, high-concurrent production serving.
  • Experience with GPU inference engines (vLLM, SGLang, Triton, TensorRT-LLM, etc.).
  • Strong background in system optimizations: batching, caching, load balancing, parallelism.
  • Low-level inference optimizations: GPU kernels, code generation.
  • Algorithmic inference optimizations: quantization, speculative decoding, distillation, low-precision numerics.
  • Experience with testing, benchmarking, and reliability of inference services.
  • Experience designing and implementing CI/CD infrastructure for inference.

Skills

C++RustGpu KernelsvLLMSglangTritonTensorrt-LlmQuantizationSpeculative DecodingCI/CDLoad BalancingDistributed Systems

Similar roles

ML Engineering jobs
xAI

Member of Technical Staff

xAIPalo Alto, CA

Build and optimize the RL training framework and infrastructure for large-scale workloads at SpaceXAI, from ablations to production runs. Requires experience with distributed systems and proficiency in Python, JAX, Rust, or C++.

180k – 440k
On-site5+ YOEML Engineering
LlamaIndex

Member of Technical Staff, Applied Research

LlamaIndexSan Francisco, CA

Build and productionize vision-language models for document understanding at LlamaIndex. Focus on training, fine-tuning, synthetic data, benchmarking, and turning research prototypes into accurate, low-latency production systems for real-world PDFs, tables, and enterprise docs. Requires 3+ years ML engineering/applied research experience with strong PyTorch skills.

180k – 250k
Hybrid3+ YOEML Engineering
Black Forest Labs

Member of Technical Staff

Black Forest LabsSan Francisco, CA

Research Engineer embedded in large-scale multimodal model training at Black Forest Labs (creators of FLUX). Own performance, stability, and low-precision optimizations for production training runs; profile, debug distributed systems, implement GPU kernels, and partner with researchers to ship frontier generative models.

180k – 290k
Hybrid7+ YOEML Engineering
xAI

Member of Technical Staff - X Search

xAIPalo Alto, CA

Develops and operates large-scale search engine infrastructure, including retrieval algorithms, indexing, and ML ranking models integrated with Grok AI. Requires experience with search systems, vector databases, and production ML in Python, Go, or Rust.

180k – 440k
On-siteML Engineering
xAI

Member of Technical Staff - Post-Training and RL

xAIPalo Alto, CA

Develops advanced post-training and reinforcement learning techniques like RLHF/DPO and reward modeling to enhance AI model reasoning, truthfulness, and real-world capabilities at xAI. Seeks passionate AI enthusiasts obsessed with truth-seeking models; prior experience preferred but not required.

180k – 600k
On-siteML Engineering