# Software Engineer, Data Infrastructure

**Company:** [Thinking Machines Lab](https://hotfix.jobs/companies/thinking-machines-lab)
**Location:** San Francisco, CA
**Role:** Data Engineering
**Salary:** $350k – $475k/yr
**Skills:** Python, Rust, Spark, Ray, Kafka, dbt, Terraform, Airflow, Delta Lake, Parquet
**Posted:** 2026-05-04

> Builds and scales data infrastructure for distributed training pipelines, multimodal data catalogs, and petabyte-scale processing systems. Collaborates with researchers using distributed systems like Spark, Ray, Kafka, and cloud data architectures. Requires backend proficiency in Python/Rust and bachelor's in CS/engineering.

## Job Description

## What You’ll Do

- Design, build, and operate scalable, fault-tolerant infrastructure for LLM Research: distributed compute, data orchestration, and storage across modalities.
- Develop high-throughput systems for data ingestion, processing, and transformation — including training data catalogs, deduplication, quality checks, and search.
- Build systems for traceability, reproducibility, and robust quality control at every stage of the data lifecycle.
- Implement and maintain monitoring and alerting to support platform reliability and performance.
- Collaborate with research teams to unlock new features, improve data quality, and accelerate training cycles.

## Skills and Qualifications

**Minimum qualifications:**
- Bachelor’s degree or equivalent experience in computer science, engineering, or similar.
- Proficiency in at least one backend language (we use **Python** or **Rust**).
- Are fluent in distributed compute frameworks such as **Apache Spark** or **Ray**.
- Are deeply familiar with cloud infrastructure, data lake architectures, and batch and streaming pipelines.
- Comfort operating across the stack and owning projects end-to-end.
- Thrive in a highly collaborative environment involving many, different cross-functional partners and subject matter experts.
- A bias for action with a mindset to take initiative to work across different stacks and different teams where you spot the opportunity to make sure something ships.

**Preferred qualifications:**
- Have hands-on experience with **Kafka**, **dbt**, **Terraform**, and **Airflow**.
- Have experience building a web crawler.
- Have extensive experience understanding and scaling deduplication, data mining, and search.
- Have strong knowledge of file formats and storage systems (e.g., **Parquet**, **Delta Lake**, etc.) and how they impact performance and scalability.
- Are proactive about documentation, testing, and empowering your teammates with good tooling.

## Logistics

**Compensation:** Depending on background, skills and experience, the expected annual salary range for this position is **$350,000 - $475,000 USD**.
**Benefits:** Thinking Machines offers generous health, dental, and vision benefits, unlimited PTO, paid parental leave, and relocation support as needed.

## Similar roles

- [Research Engineer, Economic Research](https://hotfix.jobs/jobs/5e0524fe-291d-4682-914d-4a2040886ed6) - Anthropic - San Francisco, CA - $300k – $405k/yr
- [Data Engineer, Scaling Analytics](https://hotfix.jobs/jobs/95f29a65-412b-4f98-ac4e-c97c6315ac70) - OpenAI - San Francisco, CA - $293k – $385k/yr
- [IT Controls Data Engineer](https://hotfix.jobs/jobs/d6fe5097-4a1e-473c-b90e-829663dfcd39) - OpenAI - San Francisco, CA - $293k – $385k/yr
- [Data Engineer, People Innovation Labs](https://hotfix.jobs/jobs/f1e5db56-aece-4405-b44b-0a59543295a6) - OpenAI - San Francisco, CA - $293k – $325k/yr
- [Software Engineer, Data Acquisition](https://hotfix.jobs/jobs/6279860a-f2d9-4951-9851-227bbba7cf89) - OpenAI - San Francisco, CA - $293k – $385k/yr

**Apply:** https://hotfix.jobs/jobs/f5d636b9-b692-4272-a326-a1caaacac68b
**Canonical:** https://hotfix.jobs/jobs/f5d636b9-b692-4272-a326-a1caaacac68b