Senior Software Engineer, Agent Oversight

Responsibilities

• Design and build core platform capabilities for deploying, monitoring, and evaluating agentic applications in production
• Build reliable APIs and data pipelines that capture agent telemetry, evaluation signals, and performance metrics at scale
• Work alongside ML engineers where platform work intersects with evaluation or improvement systems — bringing enough ML fluency to reason about model behavior, evaluation quality, and improvement loops while owning the software systems that make those workflows reliable
• Own the reliability, scalability, and observability of platform components serving multiple concurrent enterprise and government customers
• Work cross-functionally with product, forward deployed engineering, and customers to translate real-world deployment requirements into platform features
• Build features end-to-end: system design, implementation, debugging, and testing
• Participate in high-velocity experimentation to validate platform capabilities against real customer usage

Requirements

• 4+ years of professional software engineering experience, with strong fundamentals in backend/distributed systems, APIs, and data pipeline design
• Hands-on experience building production software for ML/LLM-powered products or platforms, such as evaluation systems, observability/monitoring, experimentation infrastructure, agent runtimes, model-serving-adjacent services, or telemetry/data pipelines
• Working knowledge of how LLM or ML systems behave in production: evaluation signals, failure modes, prompt/tool-calling workflows, experiment results, data quality issues, and the tradeoffs between offline evals and live customer behavior
• Experience partnering closely with ML engineers or applied researchers to turn prototypes, eval loops, or model-improvement workflows into reliable platform capabilities, without needing to own model training, modeling strategy, or research direction
• Experience building infrastructure or platforms that other engineering teams build on top of (internal platform, developer tools, or similar)
• Track record of taking ownership of features or components end-to-end — from design through production — within a larger platform or system
• Comfortable operating in an ambiguous, fast-changing domain where tooling and best practices are still being defined
• Strong problem-solving skills and the ability to work independently or as part of a tight-knit, cross-functional team
• Excited to work directly with ML engineers and customer-facing teams, including challenging assumptions in designs and metrics when platform behavior, model behavior, and customer needs intersect
• Gives direct, substantive feedback on designs and code, and takes it the same way — and mentors others as they grow

Nice-to-Haves

• Deep experience building or maintaining observability, monitoring, or evaluation systems for ML/LLM-powered products in production
• Familiarity with agent architectures — tool use, planning, multi-agent orchestration
• Exposure to MLOps, feature stores, model serving, or experiment infrastructure
• Experience working in regulated or enterprise contexts
• Experience reviewing others’ technical designs or mentoring engineers at a senior/staff level