# Research Engineer Intern, Evaluations

**Company:** [TensorStax](https://hotfix.jobs/companies/tensorstax)
**Location:** San Francisco, CA
**Role:** ML Engineering
**Skills:** Python, PyTorch, JAX, Reinforcement Learning, RLHF, Snowflake, BigQuery, dbt, Spark, Kafka
**Posted:** 2025-03-18

> Designs evaluation frameworks and benchmarks to test AI agents' autonomy, reasoning, and reliability in data pipelines and warehouses. Requires experience in LLM benchmarking, reinforcement learning, Python, PyTorch/JAX, and data engineering tools.

## Job Description

## What You’ll Do
- Develop evaluation environments to test AI agents' ability to reason, plan, and act autonomously within mission-critical data pipelines.
- Design benchmarks to assess model capabilities in failure detection, pipeline optimization, and agentic decision-making in data workflows.
- Implement automated assessment frameworks for language model-based agents operating over data lakes and warehouses.
- Work with synthetic and real-world datasets to create robust testing environments for AI-driven data automation.
- Collaborate with research engineers to refine reward shaping strategies, guiding models toward more efficient and agentic behaviors in data-intensive tasks.

## What We’re Looking For
- Experience in language model research, with a focus on benchmarking LLMs in mission-critical domains.
- Strong background in AI evaluation methodologies, reinforcement learning, and RLHF techniques.
- Familiarity with benchmarking language models for structured and unstructured data tasks.
- Proficiency in Python and experience with ML frameworks like PyTorch or JAX.
- Hands-on experience with data lakes, warehouses, and data engineering tools (Snowflake, BigQuery, dbt, Spark, Kafka).
- High agency—proactive, resourceful, and comfortable working in a fast-paced research environment with minimal supervision.
- Attention to detail—ability to design rigorous, reproducible experiments and evaluations.

## Bonus Points
- Contributions to open-source AI benchmarks (e.g., SweBench, BIRD, SPIDER).
- Contributions to open-source agentic frameworks.
- Experience developing custom RL environments for AI evaluation.
- Strong understanding of ETL, ELT, and data transformation pipelines.

## Similar roles

- [Software Engineer, Performance](https://hotfix.jobs/jobs/ff4b3339-6a08-4324-bcb6-635f3698a11d) - Nuro - Mountain View, CA - $153k – $175k/yr
- [Machine Learning Engineer, Relevance and Personalization](https://hotfix.jobs/jobs/e55bff4d-9948-4fd5-8f5e-51f0f5c17db4) - Airbnb - Remote - $166k – $195k/yr
- [Software Engineer](https://hotfix.jobs/jobs/823e27ee-a56e-4d3e-ac4d-dacbce6de23d) - Abridge - San Francisco, CA - $157k – $184k/yr
- [ML Engineer, Generative Video](https://hotfix.jobs/jobs/70497a05-ec28-442a-b182-4bb62ef208eb) - Mirage - New York, NY - $175k – $275k/yr
- [Software Engineer](https://hotfix.jobs/jobs/6d8b2f8e-891e-4d59-b0b0-632d28fa07ac) - Traba - New York, NY - $140k – $200k/yr

**Apply:** https://hotfix.jobs/jobs/1bb5568b-1a83-4624-a440-697d02cd6a4c
**Canonical:** https://hotfix.jobs/jobs/1bb5568b-1a83-4624-a440-697d02cd6a4c