Staff Software Engineer - AI Agent Evaluations

What You'll Do

• Define AI Quality Standards: Own the framework for how ID.me evaluates, validates, and monitors AI agents — from prompt-based features to fully autonomous multi-step workflows.
• Build Eval Infrastructure: Design and maintain evaluation pipelines for LLM outputs, agent behavior, tool use, and multi-turn interactions across development, staging, and production environments.
• Production Observability for Agents: Instrument agentic systems for behavioral drift, regression, and failure modes that traditional metrics miss — latency, correctness, hallucination rate, tool misuse, and policy adherence.
• Agentic Test Strategy: Lead the design of test suites that handle non-determinism — red-teaming agents, golden dataset construction, LLM-as-judge pipelines, and property-based testing for AI outputs.
• Champion Developer Experience: Build the internal tooling, feedback loops, and testing workflows that make it fast and safe for engineers to develop and ship AI features with confidence. Reduce friction in the agent development inner loop — local testing, fast eval runs, and clear signal on regressions.
• Drive AI-First Engineering Culture: Raise the quality bar across the engineering org by establishing patterns, tooling, and education for how teams write, test, and deploy AI features responsibly.
• Cross-Team Collaboration: Partner with Security, Platform, Product, and AI/ML teams to embed quality gates into agent development workflows.
• Mentorship: Guide senior and mid-level engineers through evaluation design, observability strategy, and testing approaches specific to AI systems.

Basic Qualifications

• Bachelor's degree in Computer Science, Engineering, or equivalent experience
• 8+ years building and operating production software systems
• Demonstrated experience evaluating or testing LLM-powered features or autonomous agents in production
• Proficiency with AI-assisted development tools (Claude Code, Cursor, or equivalent) — you build with AI every day
• Strong backend engineering fundamentals in Python, Java, Go, or equivalent
• Experience designing test infrastructure, CI/CD quality gates, or evaluation pipelines at scale
• Experience improving developer experience — building internal tooling, reducing toil, or accelerating engineering workflows
• Proven ability to lead cross-team technical initiatives and influence engineering standards
• Strong written and verbal communication across engineering, product, and leadership
• Experience building eval frameworks for LLM agents (e.g., correctness graders, LLM-as-judge, human-in-the-loop evals, benchmark dataset curation)
• Familiarity with agentic frameworks (Claude API / Anthropic SDK, BrainTrust, LangChain, LangGraph, CrewAI, or similar)
• Production monitoring experience for AI systems: behavioral drift detection, output sampling, shadow scoring
• Red-teaming or adversarial testing experience for AI models or agents

Preferred Qualifications

• Background in identity verification, fraud detection, or regulated industries
• Familiarity with Anthropic's model evaluation methodology or similar published eval research
• Experience with observability tooling (Datadog, OpenTelemetry) applied to AI workloads
• Track record of building developer tooling or platforms that other teams adopt widely

Compensation & Benefits

• Comprehensive medical, dental, vision, health savings account, flexible spending accounts (medical, limited purpose, dependent care, commuter benefit accounts)
• Basic and voluntary life and AD&D insurance
• 401(k) with company match
• Parental leave
• Unlimited paid time off subject to the terms and conditions of the PTO policy, including 8 company wide holidays
• Short and long-term disability insurance
• Accident and critical illness insurance
• Referral bonus policy, employee assistance program, pet insurance, travel assistant program
• Wellbeing and childcare discounts, benefit advocates, and a learning and development benefit