Staff + Senior Software Engineer, Cloud Inference Launch Engineering

Key Responsibilities

• Be on the critical path for frontier model launches, bringing up inference for new model architectures and shipping them to cloud platforms in lockstep with our first-party platform.
• Work with the core inference team to bring new inference features (e.g. structured sampling, prompt caching) to cloud platforms, owning the platform-specific integration that gets them to production.
• Identify and dive deep on the gaps that make inference behave differently across first-party and CSPs (config drift, observability, deployment patterns, hard cross-platform bugs) and fix them at the source.
• Design, build, and own the CI/CD infrastructure for the inference server and load balancer across cloud platforms, with shadow traffic, performance baselines (throughput and latency), and correctness checks that catch regressions before production.
• Drive down merge-to-production cycle time by making validation faster, more parallel, and cost-effective enough to run on the same constrained accelerator pool that serves customers, without trading away reliability.
• Analyze observability data across providers to identify performance bottlenecks, cost anomalies, and regressions, and drive remediation based on real-world production workloads.

Minimum Qualifications

• Strong interest in LLM serving (prior inference or ML experience not required).
• Significant software engineering experience with a strong background in high-performance, large-scale distributed systems serving millions of users.
• Track record of building automation or test infrastructure that measurably improved release velocity or reliability.
• Experience building or operating services on at least one major cloud platform (AWS, GCP, or Azure), with exposure to Kubernetes, Infrastructure as Code, or container orchestration.
• Thrive in cross-functional collaboration with both internal teams and external partners.
• Fast learner who can quickly ramp up on new technologies, hardware platforms, and provider ecosystems.
• Highly autonomous and take ownership of problems end-to-end.

Preferred Qualifications

• LLM inference optimization, batching, and caching strategies.
• Capacity-constrained scheduling or shared-resource test infrastructure.
• Solid understanding of multi-region deployments, request routing, load balancing, global traffic management.
• Working with CSP partner teams to scale infrastructure across multiple platforms, navigating differences in networking, security, privacy, and managed services.
• Proficiency in Python or Rust.

Compensation and Benefits

• Annual compensation range: $320,000–$485,000 USD (total compensation).
• Minimum education: Bachelor’s degree or equivalent.
• Location-based hybrid policy: expect staff to be in office at least 25% of the time.
• Competitive compensation, optional equity donation matching, generous vacation and parental leave, flexible working hours.