Software Engineer, Evals

Responsibilities

• Design and build large-scale evaluation pipelines measuring assistant and agent quality across thousands of real-user and synthetic workflows.
• Evaluate frontier model releases and open-source model releases, building infrastructure and quality signals to identify regressions, tradeoffs, and launch readiness.
• Build agent observability infrastructure, including trace enrichment, durable telemetry pipelines, dashboards, and debugging workflows.
• Own backend systems from architecture and design documentation through production rollout, reliability, monitoring, and iteration.
• Partner with product, machine learning, and infrastructure engineers to make evaluations a first-class part of shipping AI features.
• Connect evaluation results, customer feedback, regression analysis, and engineering workflows to concrete product improvements.
• Build systems balancing speed, reliability, enterprise security, and cost across cloud-native environments.
• Mentor engineers, raise the bar for technical design, and help shape the engineering culture of the Bangalore AI quality team.

Requirements

• 6+ years of software engineering experience building backend systems, infrastructure, distributed systems, or data platforms.
• Strong coding skills in Go, Python, Java, C++, or similar languages, with an emphasis on reliability, scale, and well-tested components.
• Experience with distributed data pipelines, production services, observability systems, or cloud-native infrastructure.
• Analytical rigor and attention to whether metrics reflect real user experience.
• Comfort working in customer-focused, cross-functional environments.
• Strong commitment to quality in both engineering systems and AI products.

Nice-to-haves

• Experience with LLM applications, evaluations, tracing, data warehouses, workflow orchestration, or machine-learning infrastructure.

Compensation & Benefits

• Compensation is determined by factors including location, level, job-related knowledge, skills, and experience.
• Certain roles may be eligible for variable compensation, equity, and benefits.