Software Engineer - BIS
As a Software Engineer on the Inference Stack team, you will build the distributed runtime that powers large-scale LLM inference. This role involves working across the stack, from developer experience to low-level infrastructure, and owning systems in production.
About the job
RESPONSIBILITIES
- Develop infrastructure and orchestration systems for deploying and managing large-scale distributed LLM inference
- Work across the stack, from customer-facing features to low-level infrastructure components
- Build platform capabilities related to routing, autoscaling, scheduling, observability, and runtime management
- Improve the reliability, scalability, and usability of our inference stack
- Collaborate closely with Model Performance engineers to make new inference optimizations broadly available to customers and easy to configure
- Help define best practices around testing, release automation, benchmarking, and operational excellence
- Debug complex production systems spanning Kubernetes, distributed runtimes, networking, and GPU workloads
- Make thoughtful engineering tradeoffs balancing performance, reliability, operational simplicity, and developer experience
- Own projects end-to-end: from architecture and implementation through deployment, monitoring, and iteration based on customer feedback
REQUIREMENTS
- Bachelor's, Master's, or Ph.D. in Computer Science, Engineering, or a related field
- Strong background in distributed systems, backend infrastructure, or platform engineering
- Experience building and operating production systems where reliability, latency, and scale are first-class concerns
- Strong sense of developer experience: you think about how systems are used, not just how they work
- Motivated and willing to learn new languages, frameworks, and systems as needed
- Ability to debug complex systems across multiple layers of the stack
- Genuine interest in inference engineering. You don’t need to have hands on experience but are willing to learn
- Excellent communication and collaboration skills
BONUS
- Experience with Kubernetes, including concepts like operators and custom resources
- Prior work on Dynamo, vLLM, SGLang, TensorRT-LLM, or similar inference frameworks
- Experience with distributed scheduling, autoscaling, or service orchestration
- Experience operating GPU workloads in production
- Familiarity with observability tooling, CI/CD systems, or release automation
- Experience contributing to open-source infrastructure or ML systems
BENEFITS
- Competitive compensation, including meaningful equity.
- 100% coverage of medical, dental, and vision insurance for employee and dependents
- Flexible PTO policy including company wide Winter Break (our offices are closed from Christmas Eve to New Year's Day!)
- Paid parental leave
- Fertility and family-building stipend through Carrot
- Company-facilitated 401(k)
- Exposure to a variety of ML startups, offering unparalleled learning and networking opportunities.
Skills
Distributed Systems, Backend Infrastructure, Platform Engineering, Kubernetes, Dynamo, vLLM, Sglang, Tensorrt-Llm, Gpu Workloads, CI/CD
Similar jobs
ML Engineering jobs
Mirage
New York, NY
Build and deploy agentic systems that power AI-driven creative video workflows. The role requires 5+ years of experience, production ML or agentic pipeline development, context engineering, and expertise in evaluation and agent infrastructure.
Mirage
New York, NY
Build and advance agentic machine-learning systems for multimodal creative tasks, with a focus on video understanding, reasoning, control, and tool use. The role requires strong production ML or agent-pipeline experience and deep knowledge of modern LLM techniques.
Taste Labs
San Francisco, CA
Build evaluation methods, RL environments, agent tooling, and scalable infrastructure that make subjective qualities such as design and taste measurable for frontier AI models. The role requires experience with evaluations, RL environments, ML or post-training, plus strong backend engineering skills.
Mirage
New York, NY
Build and scale generative video and multimodal models, optimizing training and inference for efficiency, throughput, and ultra-low latency. The role requires deep learning systems expertise, strong PyTorch/CUDA experience, and the ability to move research models into production.
Earnin
Mountain View, CA
Build production-grade AI agents, evaluation infrastructure, and developer tooling that make AI-assisted engineering faster, safer, and reusable across teams. The role requires software engineering experience, platform or internal developer-product experience, and hands-on expertise with LLM integration and orchestration.