# Research Engineer - Evaluations

**Company:** [Luma AI](https://hotfix.jobs/companies/lumalabs-ai)
**Location:** Palo Alto, CA
**Role:** AI Research
**Experience:** 5+ years
**Skills:** Python, PyTorch, JAX, TensorFlow, Machine Learning, Generative Models, Diffusion Models, LLMs, Multimodal Models, CI/CD, Data Pipelines, Distributed Systems, Reinforcement Learning, Reward Modeling
**Posted:** 2025-08-26

> Designs and scales infrastructure for evaluating multimodal generative AI models, including pipelines, metrics, and integration into training loops. Requires 5+ years in ML evaluation systems, expertise in visual data, Python, and ML frameworks like PyTorch.

## Job Description

## Responsibilities
- Design and implement scalable pipelines for automated evaluation of generative models, with a focus on visual and multimodal outputs (image, video, text, audio).
- Develop novel metrics and evaluation models that capture qualities like fidelity, coherence, temporal consistency, and alignment with human intent.
- Integrate evaluation signals into training loops (including reinforcement learning and reward modeling) to continuously improve model performance.
- Build infrastructure for large-scale regression testing, benchmarking, and monitoring of multimodal generative models.
- Collaborate with researchers running human studies to translate human evaluation frameworks into automated or semi-automated systems.
- Partner with model researchers to identify failure cases and build targeted evaluation harnesses.
- Maintain dashboards, reporting tools, and alerting systems to surface evaluation results to stakeholders.
- Stay current with emerging evaluation techniques in generative AI, multimodal LLMs, and perceptual quality assessment.

## Qualifications
- Master's or PhD in Computer Science, Machine Learning, or a related technical field (or equivalent industry experience).
- 5+ years of experience building ML evaluation systems, model pipelines, or large-scale infrastructure.
- Hands-on experience working with visual data (images and/or video), including evaluation, modeling, or data preparation.
- Proficiency in Python and ML frameworks (PyTorch, JAX, or TensorFlow).
- Familiarity with human-in-the-loop evaluation workflows and how to scale them with automation.
- Strong background in machine learning, with experience in generative models (diffusion, LLMs, multimodal architectures).
- Strong software engineering skills (CI/CD, testing, data pipelines, distributed systems).

## Nice to Have
- Experience with reinforcement learning or reward modeling.
- Prior work on perceptual metrics, multimodal evaluation benchmarks, or retrieval-based evaluation.
- Background in large-scale model training or evaluation infrastructure.
- Experience designing metrics for perceptual quality.
- Familiarity with creative media workflows (film, VFX, animation, digital art).
- Contributions to open-source evaluation libraries or benchmarks.

## Similar roles

- [ML Scientist](https://hotfix.jobs/jobs/401a3fda-7a0b-4af9-aa9c-134e4ae4b19c) - Sesame - San Francisco, CA - $190k – $320k/yr
- [Research Scientist](https://hotfix.jobs/jobs/34963160-7769-44e3-8e41-872372a811ed) - OpenRouter - Remote
- [Research Engineer, Benchmarks](https://hotfix.jobs/jobs/71636332-204e-4abc-b0b5-c2b5c2ca3726) - hud - San Francisco, CA
- [Research Scientist](https://hotfix.jobs/jobs/5c6acbfc-713c-4c38-bb2c-7a8d9dbc74d1) - Spotify - Remote
- [Research Scientist, Life Sciences](https://hotfix.jobs/jobs/3612b705-0fb1-4f66-a4a1-0b28ba382b5c) - Anthropic - San Francisco, CA - $300k – $320k/yr

**Apply:** https://hotfix.jobs/jobs/1728355f-0f25-421a-98e4-e1d713ba3351
**Canonical:** https://hotfix.jobs/jobs/1728355f-0f25-421a-98e4-e1d713ba3351