Senior Test Engineer - Metrics Quality

Responsibilities

• Own acceptance and release qualification for new and changed metrics used in autonomous-driving software decisions.
• Define and maintain comprehensive metric case spaces, including situations where metrics should remain silent.
• Identify silent failures and assess how often metrics are incorrect or fail to report.
• Review metric implementations against expected cases to find unhandled scenarios.
• Build or source test data for required metric situations and improve data-generation workflows.
• Present findings to metric owners with concrete examples, impact, and scale.
• Evaluate whether metrics reveal aggregate degradation across recorded-scene datasets.
• Run and maintain regression testing that detects changes in metric meaning and behavior, while improving release efficiency.

Requirements

• 6+ years of software testing or test engineering experience, with depth in data-heavy or analytical systems.
• Strong Python and SQL skills, including analysis scripts and independent reference computations.
• Statistical literacy, including distributions, variance, sample size, aggregation pitfalls, and distinguishing noise from meaningful differences.
• Product-oriented approach to determining what metrics are intended to measure and support.
• Ability to communicate and defend findings with evidence.
• Fluent use of LLMs for analysis, test-data generation, unfamiliar-code review, and reasoning validation.
• Clear written English.

Nice to Have

• Experience with autonomous vehicles, robotics, or machine-learning system evaluation.
• ClickHouse or a similar analytical database.
• Experience testing evaluation pipelines or A/B testing infrastructure.
• Experience designing human-reviewer annotation tasks and measuring reviewer agreement.
• Knowledge of US road rules and driver behavior.