Applied AI Researcher, Agent Systems & Evaluation

Responsibilities

• Build rigorous closed-loop evaluation systems for AI agents operating within engineering workflows.
• Improve agent performance against real-world production data, codebases, infrastructure, and engineer requests rather than relying solely on public benchmarks.
• Own evaluation pipelines end to end, including:
  • Collecting ground truth from production traces and human accept/reject/edit signals.
  • Building representative task suites and assessing when model-based judges are trustworthy.
  • Establishing noise floors, statistical acceptance standards, anti-gaming safeguards, and experiments for production environments.
• Automate hill climbing across prompts, context strategies, tool sets, routing, reasoning budgets, and model selection once evaluation loops are trustworthy.
• Post-train open-source vision-language models using supervised fine-tuning and reinforcement learning with proprietary autonomous-driving data.
• Define data requirements and work with labeling resources to create targeted training and evaluation datasets.
• Monitor research developments in frontier AI and translate relevant findings into experiments against production workloads.
• Investigate test-time scaling, including reasoning budgets, sampling and search strategies, verifier-guided selection, escalation, and stopping policies.
• Quantify real-world impact and determine whether measured results support claimed improvements.
• Partner closely with engineering to ensure evaluation systems run safely and reliably at scale.

Requirements

• Engineering background with strong research judgment.
• Ability to design, implement, and ship AI research systems.
• Fluency in frontier models and model development.
• Expertise in evaluating foundation models and agent systems composed of models, tools, memory, retries, verifiers, and human feedback.
• Ability to establish trustworthy measurements and interpret statistical evidence.
• Experience working with production data and ambiguous real-world tasks.
• Hands-on experience with supervised fine-tuning and reinforcement learning for open-weight models.

Nice-to-haves

• Experience with vision-language models.
• Experience with autonomous-driving or other real-world operational data.
• Experience designing labeling programs and task-specific datasets.
• Experience with automated optimization or hill-climbing systems.
• Experience with test-time scaling, inference-time search, verifiers, or model routing.