Skip to content
CursorCursorNew York, NY

Software Engineer, Model Routing & Inference

Build the inference platform powering all AI interactions, focusing on high-throughput, low-latency distributed systems for model routing, failover, and cost/performance optimization at massive scale. Requires strong engineering fundamentals for production systems handling millions of requests.

Salary not listed
On-siteBackend Engineering

About the role

Responsibilities

  • Build and evolve the inference gateway, a single abstraction over every provider's API semantics, so model onboarding becomes a config change.
  • Design intelligent cross-provider failover so no single provider outage causes user-visible degradation.
  • Design routing backpressure and admission control so traffic spikes don't cascade into providers.

Requirements

  • Deep experience building high-throughput, low-latency distributed systems, especially in inference serving, traffic routing, or real-time data pipelines.
  • Comfortable reasoning about cost/performance tradeoffs at scale (GPU utilization, provider economics, capacity planning).
  • Strong software engineering fundamentals and enjoy shipping production systems that handle millions of requests.
  • Make good calls in the gray area: weighing reliability, cost, latency, and user experience when there isn't a single "right" answer.

Skills

Distributed SystemsInference ServingTraffic RoutingReal-Time Data PipelinesGpu UtilizationApi GatewaysFailover SystemsBackpressureAdmission ControlCapacity Planning
Front

Software Engineer (Backend), Deliverability

FrontSan Francisco, CA

Technical lead on the Deliverability team owning high-volume email infrastructure, ingestion, routing, and outbound delivery systems that process millions of events. Requires 5+ years building scalable distributed backend systems with strong ownership of architecture, reliability, and on-call duties.

178k – 204k/yr
Hybrid5+ YOEBackend Engineering
Scribe

Backend Engineer

ScribeSan Francisco, CA

Backend Engineer building and optimizing core document workflows (viewing, editing, embeds, exports) and greenfield projects in a Django monolith. Requires 3+ years backend experience, strong Python skills, performance optimization, and automated testing.

140k – 185k/yr
Hybrid3+ YOEBackend Engineering
OpenAI

Software Engineer, Codex -Enterprise Controls

OpenAISan Francisco, CA

Build and operate the enterprise control plane for OpenAI's Codex AI software engineer, focusing on security, governance, identity, access controls, encryption, and policy enforcement systems for large organizations.

230k – 385k/yr
Hybrid5+ YOEBackend Engineering
ZoomInfo

Software Engineer III

ZoomInfoBethesda, MD +1

Software Engineer III designing, building, and maintaining scalable backend systems with Java, Node.js/NestJS, Kafka, and cloud technologies (AWS/GCP). Requires Master's in CS + 2 years (or Bachelor's + 4 years) backend experience, microservices, databases, and REST APIs.

112k – 176k/yr
Hybrid4+ YOEBackend Engineering
Upstart

Software Engineer, Upstart Bank

UpstartUnited States

Software Engineer building scalable backend systems, services, and APIs for Upstart Bank, a modern AI-enabled national bank. Translate regulatory and business requirements into reliable software in a fintech environment; 4+ years backend/full-stack experience required.

14k – 187k/yr
Remote4+ YOEBackend Engineering