# Software Engineer, Evaluation Platform / Infra

**Company:** [Thinking Machines Lab](https://hotfix.jobs/companies/thinking-machines-lab)
**Location:** San Francisco, CA
**Role:** Fullstack Engineering
**Salary:** $300k – $475k/yr
**Experience:** 2+ years
**Skills:** Python, Rust, React, TypeScript, Distributed Systems, Data Pipelines, APIs, Databases, Workflow Orchestration, LLMs, Multimodal Models, Observability
**Posted:** 2026-08-26

> Build an end-to-end evaluation platform for frontier AI models, spanning backend systems, data pipelines, APIs, and user-facing applications. The role requires software engineering experience, model evaluation expertise, and the ability to collaborate closely with researchers.

## Job Description

## Responsibilities
- Design, build, and maintain a platform for authoring, running, tracking, and analyzing model evaluations used in daily research and model releases.
- Work across evaluation libraries, distributed backend systems, data pipelines, APIs, and user-facing applications.
- Build flexible abstractions for evaluation tasks, environments, graders, datasets, and model outputs.
- Ensure evaluation results are reproducible and trustworthy through versioning, provenance, observability, failure recovery, and quality controls.
- Partner with researchers to identify bottlenecks and turn bespoke workflows into self-serve systems.
- Collaborate with pre-training, post-training, applied, and research tooling teams.

## Requirements
- Bachelor's degree or equivalent practical experience in computer science, engineering, machine learning, or a related field.
- 2+ years of post-graduate software engineering or machine learning engineering experience, excluding internships.
- Hands-on experience building or maintaining evaluations, benchmarks, graders, or model-quality systems for large language or multimodal models.
- Strong software engineering fundamentals and experience building reliable, maintainable systems.
- Proficiency in at least one backend programming language; Python and Rust are primarily used.
- Experience with databases, data pipelines, distributed systems, or other data-intensive infrastructure.
- Ability to work across the stack and own projects from discovery through deployment and operation.
- Experience collaborating with cross-functional partners and subject-matter experts.

## Nice-to-haves
- Experience building frameworks, SDKs, or developer tools with thoughtful abstractions and strong user experience.
- Experience with distributed job execution, workflow orchestration, sandboxed environments, or large-scale data processing.
- Experience building interfaces for inspecting complex data, comparing experiments, or debugging model behavior.
- Familiarity with large language or multimodal model evaluation, including model-based grading, human evaluation, or synthetic data.
- Experience working closely with researchers to turn rapidly evolving needs into durable systems.
- Experience at a startup or on a small team building technically complex products end to end.

## Compensation and Benefits
- Expected annual salary: $300,000–$475,000 USD, depending on background, skills, and experience.
- Health, dental, and vision benefits.
- Unlimited paid time off.
- Paid parental leave.
- Relocation support as needed.
- Visa sponsorship.

## Similar jobs

- [Software Engineer, Research Tools](https://hotfix.jobs/jobs/27d7e638-a482-4872-96ba-c556f15c2363) - Thinking Machines Lab - San Francisco, CA - $300k – $475k/yr
- [Software Engineer III](https://hotfix.jobs/jobs/1c26a526-4463-44c7-be72-40c8136c55e5) - Garner Health - New York, NY - $194k – $220k/yr
- [Software Engineer](https://hotfix.jobs/jobs/62cb0672-24a7-4902-99dd-fc29a772abb2) - Applied Intuition - Sunnyvale, CA - $183k – $244k/yr
- [Software Engineer Intern](https://hotfix.jobs/jobs/f3388a5f-2bc9-4b50-8f3c-58a8961c5ddd) - The Voleon Group - Berkeley, CA - $175k – $175k/yr
- [Generalist Engineer](https://hotfix.jobs/jobs/f84ad5b6-0def-4d6e-ac70-f8ac7e334495) - Greptile - San Francisco, CA - $170k – $300k/yr

**Apply:** https://hotfix.jobs/jobs/b7c4378f-cce3-4101-85d9-fd087b4bf3a3
**Canonical:** https://hotfix.jobs/jobs/b7c4378f-cce3-4101-85d9-fd087b4bf3a3