Backend Software Engineer (Evals)
Build evals infrastructure and backend services for OpenAI's support automation, focusing on reliable pipelines, monitoring, and AI model integration. Requires 4+ years backend experience with Python, FastAPI, Postgres, and ML/LLM evals.
About the job
In this role, you will:
- Design eval pipelines that are reliable, reproducible, and extendable
- Build infrastructure for continuous eval monitoring frameworks (regression/drift monitoring, robust golden datasets) with feedback loops
- Design, build, and maintain backend services and APIs for intelligent automation and knowledge systems
- Integrate and structure data across internal platforms for downstream systems and AI workflows
- Collaborate with data, research, and engineering teams to integrate OpenAI models into workflows
- Own the full development lifecycle of new backend systems
- Build with scale and maintainability in mind while iterating rapidly
You might be a great fit if you have:
- 4+ years of backend engineering experience at product-driven companies (excluding internships)
- Proficiency in backend technologies: Python, FastAPI, Postgres
- Experience designing and scaling distributed systems, APIs, or data processing pipelines
- Experience building AI agents or applications, including evals and performance improvement via prompting or scaffolding
- Familiarity with LLM evaluation methods and patterns like multi-agent workflows, tool use, or long context
- Experience creating production evals and/or measuring ML/LLM model performance at scale
- Pragmatic mindset for iterative shipping toward long-term vision
Skills
Python, FastAPI, Postgres, Distributed Systems, APIs, Data Processing Pipelines, Llm Evals, AI Agents, Multi-Agent Workflows, Ml Models
Similar jobs
Backend Engineering jobs
OpenAI
San Francisco, CA
Build and operate backend and financial infrastructure supporting pricing, payments, billing, subscriptions, entitlements, and invoicing. The role requires 5+ years of software engineering experience with distributed systems, transactional workflows, and highly reliable, auditable platforms.
Stripe
Seattle, WA
Build and operate scalable backend services, APIs, and real-time data systems while contributing to architecture, reliability, and product-driven solutions. Requires a relevant master’s degree with 3 years of experience or a bachelor’s degree with 5 years, plus broad distributed systems and programming expertise.
Fluidstack
San Francisco, CABuild foundational infrastructure platforms and operational tooling for global data center operations, including CMDB, DCIM, asset management, automation, and observability systems. Requires 3+ years of production software development experience and strong Python or Go skills.
Rain
Remote
Design and lead secure, scalable wallet infrastructure spanning custody, key management, signing, authorization, and recovery. The role requires deep expertise in wallet or cryptographic security infrastructure, distributed systems architecture, and modern blockchain account models.
Baseten
San Francisco, CABuild and own Baseten’s identity and authorization platform, including fine-grained permissions, credential systems, and enterprise administration. The role requires backend systems experience, production authorization expertise, and the ability to operate secure multi-tenant systems at scale.