Skip to content
HarveyHarveyNew York, NY

Software Engineer, AI Platform

Builds AI platform foundation including model routing, agent infrastructure, context management, and evaluation systems for legal AI products. Requires 5+ years backend experience with 1+ year AI/ML, production multi-model systems, and shared platform expertise.

220k – 300k/yr
On-site5+ YOEBackend Engineering

About the role

Representative Projects

  • Context Engineering & Agent Infrastructure: Build platform-level systems for context management, session state, and memory.
  • Model Integration & Routing: Own infrastructure for onboarding foundation models and routing to the right one.
  • Evaluation Infrastructure: Build shared eval tooling and frameworks.
  • Shared Abstractions: Create SDKs, platform primitives, and developer tooling.

What You’ll Do

  • Design and build abstractions and platform-level systems that improve all agentic products.
  • Own infrastructure for model integration, routing, and evaluation.
  • Build evaluation frameworks and tooling for AI quality iteration.
  • Partner with product engineering teams, PMs, and design to launch AI products.
  • Evaluate, prototype, and integrate latest AI and agentic systems advancements.

What You Have

  • 5+ years building backend systems, with 1+ year in AI/ML engineering (Staff: 8+ years, technical leadership).
  • Experience shipping multi-model or multi-provider AI systems in production.
  • Familiarity with context management, session state, or memory in AI/distributed systems.
  • Track record building internal platforms, SDKs, or shared infrastructure adopted by teams.
  • Strong judgment on abstractions; pragmatic about shipping.
  • Excitement for agentic AI infrastructure challenges.
  • Bias toward full ownership in ambiguity.

Bonus: Evaluation frameworks, agent/function-calling, legal domain, early-stage/hyper-growth experience.

Compensation

$220,000 - $300,000

Skills

AI/MLBackend SystemsModel RoutingContext ManagementAgent InfrastructureEvaluation FrameworksSDKsDistributed SystemsFoundation ModelsFunction-Calling

Similar roles

Backend Engineering jobs
Hex

Software Engineer, Foundations

HexSan Francisco, CA +1

Builds and evolves core backend infrastructure including APIs, realtime transport, storage frameworks, and job orchestration systems at Hex. Requires 5+ years backend experience with TypeScript/Node.js, SQL, and shared systems expertise.

215k – 270k/yr
Remote5+ YOEBackend Engineering
OpenAI

Software Engineer, Codex -Enterprise Controls

OpenAISan Francisco, CA

Build and operate the enterprise control plane for OpenAI's Codex AI software engineer, focusing on security, governance, identity, access controls, encryption, and policy enforcement systems for large organizations.

230k – 385k/yr
Hybrid5+ YOEBackend Engineering
OpenAI

Software Engineer, Private Computing

OpenAISan Francisco, CA

Design, build, and scale privacy features and infrastructure for ChatGPT, API, and consumer devices using confidential computing and encryption. Requires 5+ years of software engineering experience and production experience with trusted execution environments.

230k – 385k/yr
Hybrid5+ YOEBackend Engineering
OpenAI

Software Engineer, Enterprise AI Platform

OpenAISan Francisco, CA

Build internal AI-native apps, MCP connectors, multi-agent workflows, and reusable platform components for enterprise operations across Finance, People, and GTM. Strong Python, system design, enterprise integrations, and applied AI experience required.

230k – 385k/yr
On-siteBackend Engineering
Glean

Software Engineer, APIs & Context Platform

GleanMountain View, CA

Build and evolve Glean's REST APIs, API gateway, and developer SDKs for their AI platform. Requires 5+ years experience building public-facing APIs and strong REST design skills.

210k – 260k/yr
Hybrid5+ YOEBackend Engineering