# Engineer, Supercomputing & Distributed Systems

**Company:** [Krea](https://hotfix.jobs/companies/krea)
**Location:** San Francisco, CA
**Role:** DevOps / SRE
**Skills:** Python, Kubernetes, PyTorch, Duckdb, Pyarrow, InfiniBand, Nccl, Rdma, SQL, ETL, Arrow, pandas, NumPy, Kafka, Pulsar
**Posted:** 2026-04-03

> Builds and operates supercomputing infrastructure for AI research including 1000+ GPU Kubernetes clusters, distributed data pipelines processing petabytes, and fault-tolerant training systems. Requires strong distributed systems intuition and experience with Python, PyTorch, and large-scale infrastructure.

## Job Description

## Responsibilities

### Distributed data systems
- Design multi-stage pipelines that turn petabytes of raw data into clean, annotated datasets
- Run classification models on billions of images
- Deploy and combine LLMs to caption massive multimedia data

### GPU infrastructure
- Manage distributed training and inference on 1000+ GPU Kubernetes clusters
- Solve orchestration and scaling for large-scale GPU job processing
- Scale workloads and research between clusters in multiple datacenters

### Distributed training
- Profile and optimize dataloaders streaming thousands of images per second
- Profile and debug InfiniBand networking on huge training runs
- Build fault tolerance systems for large-scale pretraining
- Collaborate with researchers on evolving RL infrastructure

### Applied ML pipelines
- Find clean scenes in millions of videos using distributed shot-boundary detection
- Customize and train models to filter billions of images for questions like "is this a screenshot?"
- Build the systems that bridge raw cluster capacity and research output

## Requirements
- Intuition for distributed systems and great mental model of how systems interact under different conditions
- Work heavily with **Python**, **Kubernetes**, **Torch**, and data tools like **DuckDB**, **Arrow**

Strong candidates may have experience with:
- **Python**, **PyArrow**, **DuckDB**, **SQL**, massive relational databases, **PyTorch**, **Pandas**, **NumPy**
- **Kubernetes**
- Designing and implementing large-scale ETL systems
- Fundamental knowledge of containerization, operating systems, file-systems, and networking
- Distributed systems design
- Distributed training systems (**NCCL**, **InfiniBand**, **RDMA**)
- Streaming and event processing systems (**Kafka**, **Pulsar**, or similar)
- **PyTorch** internals, custom dataloaders, and training infrastructure

## Similar roles

- [Software Engineer, Platform & Infrastructure](https://hotfix.jobs/jobs/2a4d6818-4e00-4a08-8339-aba841ed1ada) - Forus - New York, NY
- [Software Engineer, Developer Productivity](https://hotfix.jobs/jobs/3d3eb78d-cb73-4b62-bef0-6c84085fdf5a) - Forus - New York, NY
- [Site Reliability Engineer](https://hotfix.jobs/jobs/8def48bc-b809-4397-bd55-28b9747c1f82) - Runloop - San Francisco, CA
- [Build & Release Engineer](https://hotfix.jobs/jobs/c41c87c5-6007-46e9-8648-635789c39b4a) - Applied Intuition - Sunnyvale, CA - $118k – $200k/yr
- [Software Engineer, Async Platform](https://hotfix.jobs/jobs/f1db6fa3-5b80-4939-86b0-39e6f0db0694) - Lyft - Seattle, WA - $118k – $147k/yr

**Apply:** https://hotfix.jobs/jobs/20a9f124-b9b1-4979-9743-5acc079bd3cf
**Canonical:** https://hotfix.jobs/jobs/20a9f124-b9b1-4979-9743-5acc079bd3cf