# Software Engineer, Distributed Data Systems

**Company:** [Exa](https://hotfix.jobs/companies/exa)
**Location:** San Francisco, CA
**Role:** Data Engineering
**Salary:** $180k – $350k/yr
**Skills:** Delta Lake, Iceberg, Hudi, Kafka, Flink, Ray, Spark, ClickHouse, Lance, Rapids, Cudf
**Posted:** 2025-12-19

> Architects and builds massive-scale data infrastructure for web crawling, embedding model training, and real-time search, handling hundreds of petabytes. Requires expertise in lakehouse architectures, distributed processing pipelines, and streaming systems like Kafka and Flink.

## Job Description

## Responsibilities
- Architect and build data infrastructure for crawling billions of pages, training embedding models, and serving real-time search.
- Design systems that scale to hundreds of petabytes.
- Example projects: Design lakehouse architecture for 100+ PB web crawl data; build streaming pipelines processing billions of documents per day; architect data layer for embedding training on Ray; scale ClickHouse for petabyte-scale analytical queries.

## Requirements
- Deep understanding of lakehouse architectures (**Delta Lake**, **Iceberg**, **Hudi**) and when to use them.
- Experience building and operating large-scale distributed data processing pipelines.
- Hands-on experience with streaming data systems (**Kafka**, **Flink**, or similar).
- Familiarity with **Ray**, **Spark**, or **ClickHouse** at production scale.
- Obsessive focus on reliability and building systems that don't page you at 3am.

## Nice-to-Haves
- Experience with **Lance** or other vector-native storage formats.
- Background in GPU-accelerated data processing (**RAPIDS**, **cuDF**).

## Similar roles

- [Data Engineer](https://hotfix.jobs/jobs/6b412eb4-37c6-4c36-8f77-10e86bc79c93) - Actively AI - New York, NY - $180k – $210k/yr
- [Data Engineer](https://hotfix.jobs/jobs/7443d425-2ed8-49a3-9b59-ee77171cd7d0) - Baseten - San Francisco, CA - $180k – $250k/yr
- [Software Engineer - Data Platform](https://hotfix.jobs/jobs/f6588639-59ab-484a-9fab-418122ada203) - xAI - Palo Alto, CA - $180k – $440k/yr
- [Software Engineer, Data Infrastructure](https://hotfix.jobs/jobs/cfc2b08d-e49a-4d20-a3f4-2f4fffbd8b88) - Datology AI - Redwood City, CA - $180k – $300k/yr
- [Data Engineer](https://hotfix.jobs/jobs/d5dae85f-a730-436c-a9e7-adb1ca41470d) - Airtable - San Francisco, CA - $180k – $222k/yr

**Apply:** https://hotfix.jobs/jobs/515ee00c-9aa9-4052-8be3-d18af7f2ea6a
**Canonical:** https://hotfix.jobs/jobs/515ee00c-9aa9-4052-8be3-d18af7f2ea6a