# Research Engineer, Infrastructure

**Company:** [Cognition](https://hotfix.jobs/companies/cognition)
**Location:** San Francisco, CA
**Role:** DevOps / SRE
**Skills:** Python, C++, PyTorch, Distributed Systems, Gpu Programming, Parallelism, Data Pipelines, Experiment Orchestration, Performance Profiling, Networking
**Posted:** 2026-04-08

> Builds and owns distributed training infrastructure, experiment orchestration, data pipelines, and performance optimizations for large-scale AI research on GPU clusters. Requires deep systems expertise, Python/C++/PyTorch proficiency, and ML understanding to accelerate frontier research.

## Job Description

## Responsibilities
- Build and own distributed training infrastructure for large-scale GPU clusters, including job launchers, checkpointing, recovery, fault tolerance, and monitoring.
- Own infrastructure for scaling agent rollouts in VM sandboxes at RL training scales.
- Profile and optimize training throughput: data loading, communication, memory, compute efficiency to improve step time and MFU.
- Design experiment orchestration and tooling to launch, track, and analyze experiments.
- Build high-throughput, reliable data pipelines for training and evaluation.
- Debug and resolve training failures across GPUs, networking, numerics, and data.
- Implement and optimize parallelism strategies: data, tensor, pipeline, sequence.
- Anticipate research needs and build scaling infrastructure proactively.

## Requirements
- Deep experience building/operating distributed training systems for large models.
- Strong systems engineering: distributed systems, networking, storage, performance reasoning.
- Proficiency in **Python**, **C++**; systems-level **PyTorch** or equivalent.
- Hands-on GPU performance profiling, memory optimization, compute efficiency.
- Experience with parallelism strategies for large model training.
- Track record building tooling to accelerate research workflows.
- Strong debugging in complex distributed systems.
- ML knowledge to engage with researchers.

## Similar roles

- [Software Engineer, Platform & Infrastructure](https://hotfix.jobs/jobs/2a4d6818-4e00-4a08-8339-aba841ed1ada) - Forus - New York, NY
- [Software Engineer, Developer Productivity](https://hotfix.jobs/jobs/3d3eb78d-cb73-4b62-bef0-6c84085fdf5a) - Forus - New York, NY
- [Site Reliability Engineer](https://hotfix.jobs/jobs/8def48bc-b809-4397-bd55-28b9747c1f82) - Runloop - San Francisco, CA
- [Build & Release Engineer](https://hotfix.jobs/jobs/c41c87c5-6007-46e9-8648-635789c39b4a) - Applied Intuition - Sunnyvale, CA - $118k – $200k/yr
- [Software Engineer, Async Platform](https://hotfix.jobs/jobs/f1db6fa3-5b80-4939-86b0-39e6f0db0694) - Lyft - Seattle, WA - $118k – $147k/yr

**Apply:** https://hotfix.jobs/jobs/9806122c-1437-49fe-87f5-4dbbf01ad5bc
**Canonical:** https://hotfix.jobs/jobs/9806122c-1437-49fe-87f5-4dbbf01ad5bc