# Software Engineer - Benchmarking

**Company:** [Office Hours](https://hotfix.jobs/companies/office-hours)
**Location:** Remote
**Role:** ML Engineering
**Salary:** $160k – $210k/yr
**Experience:** 4+ years
**Skills:** Python, Docker, React, Next.js, Tailwind, PyTorch, Hugging Face, Model Apis, Llm Evaluation, Benchmarking, Data Pipelines, GitHub
**Posted:** 2026-08-17

> Build reproducible systems for AI model benchmarking, including datasets, evaluation pipelines, containerized environments, scoreboards, and analysis tools. The role requires 4+ years of professional engineering experience, strong Python, dataset rigor, and Docker expertise.

## Job Description

## Responsibilities
- Prepare and maintain benchmark datasets, including cleaning, preparation, conversion into runnable formats, and ongoing maintenance.
- Validate that tasks are complete, consistent, and executable, and flag ambiguities that could compromise results.
- Build and maintain evaluation pipelines across model APIs and terminal agents to ensure reproducible and comparable results.
- Create lightweight, containerized evaluation environments and viewers for tasking and tool-use evaluation.
- Support fine-tuning of small open-source large language models and compare baseline with post-training performance.
- Develop published scoreboards and leaderboards covering model-, benchmark-, task-, domain-, and rubric-level performance.
- Build analysis tools to identify recurring failure modes, compare models and agent scaffolds, and track capability improvements and regressions.
- Build tooling for expert annotation and data-generation projects, including HTML viewers and internal tools for task authoring, review, quality control, and structured data collection.
- Collaborate with researchers and engineers to ensure evaluation data and outputs are accurate, consistent, and integrated into published results.

## Requirements
- 4+ years of professional experience building and maintaining complex systems.
- Strong Python skills and the ability to write robust, maintainable code.
- Experience preparing, cleaning, and maintaining datasets.
- Experience with Docker and reproducible execution environments.
- Ability to collaborate with researchers and scientists and translate methodology into working systems.

## Nice-to-haves
- Experience running AI evaluations or using frameworks such as Harbor, Terminal-Bench, or Inspect.
- Experience fine-tuning or post-training open-source large language models.
- Experience with agentic, multi-turn, long-context, or tool-use evaluation.
- Experience validating LLM-as-judge or rubric-based grading setups.
- Background or strong interest in a scientific or technical domain.
- Experience building data-heavy dashboards, leaderboards, or visualizations.
- Open-source contributions or published work related to benchmarks and measurement.

## Tech Stack
- **Evaluations:** Python, model APIs, agent and evaluation frameworks, custom evaluation tooling
- **Models:** Major AI provider APIs, terminal agents, Hugging Face ecosystem, PyTorch
- **Environments:** Docker
- **Publishing:** React, Next.js, Tailwind
- **Workflow:** GitHub, Slack, Notion, Linear

## Compensation and Benefits
- Base salary range: **$160,000–$210,000**, based on seniority, relevant experience, and location.
- Equity.
- Medical, dental, and vision coverage.
- 401(k).
- Monthly wellness and fitness stipend.
- Paid time off and company holidays.
- Annual company off-sites.
- Parent-friendly policies, remote flexibility, and paid family leave.

## Similar jobs

- [Research Scientist II](https://hotfix.jobs/jobs/4ac4da1a-a497-413d-b8fa-6b3b58cef7e1) - Pindrop - Remote - $160k – $185k/yr
- [Software Engineer - Prediction and Planning ML](https://hotfix.jobs/jobs/21b9c778-e1ae-4695-b26d-fec68ea8a8cc) - Applied Intuition - Sunnyvale, CA - $151k – $258k/yr
- [AI Engineer, Enablement](https://hotfix.jobs/jobs/6ae315a1-a605-45dc-b1ee-88fa8e9dee24) - LangChain - New York, NY - $150k – $195k/yr
- [Member of Technical Staff — Frontier Data](https://hotfix.jobs/jobs/20efe17f-aa7c-401b-8add-7086d4571fba) - Roboflow - Remote - $150k – $300k/yr
- [Algorithm Engineer](https://hotfix.jobs/jobs/3bef67e8-d4e7-4866-a9b8-6c7863fe2e96) - Beacon Biosignals - Remote - $150k – $170k/yr

**Apply:** https://hotfix.jobs/jobs/b7804dfa-115e-4c6d-9080-36b06fd557b8
**Canonical:** https://hotfix.jobs/jobs/b7804dfa-115e-4c6d-9080-36b06fd557b8