Skip to content
ExaExaSan Francisco, CA

Software Engineer, Distributed Data Systems

Architects and builds massive-scale data infrastructure for web crawling, embedding model training, and real-time search, handling hundreds of petabytes. Requires expertise in lakehouse architectures, distributed processing pipelines, and streaming systems like Kafka and Flink.

180k – 350k/yr
On-siteData Engineering

About the role

Responsibilities

  • Architect and build data infrastructure for crawling billions of pages, training embedding models, and serving real-time search.
  • Design systems that scale to hundreds of petabytes.
  • Example projects: Design lakehouse architecture for 100+ PB web crawl data; build streaming pipelines processing billions of documents per day; architect data layer for embedding training on Ray; scale ClickHouse for petabyte-scale analytical queries.

Requirements

  • Deep understanding of lakehouse architectures (Delta Lake, Iceberg, Hudi) and when to use them.
  • Experience building and operating large-scale distributed data processing pipelines.
  • Hands-on experience with streaming data systems (Kafka, Flink, or similar).
  • Familiarity with Ray, Spark, or ClickHouse at production scale.
  • Obsessive focus on reliability and building systems that don't page you at 3am.

Nice-to-Haves

  • Experience with Lance or other vector-native storage formats.
  • Background in GPU-accelerated data processing (RAPIDS, cuDF).

Skills

Delta LakeIcebergHudiKafkaFlinkRaySparkClickHouseLanceRapidsCudf

Similar roles

Data Engineering jobs
Actively AI

Data Engineer

Actively AINew York, NY

Own data and analytics end-to-end: architect internal systems, build metrics/dashboards, and translate customer and product signals into structured inputs for AI agents.

180k – 210k/yr
On-siteData Engineering
Baseten

Data Engineer

BasetenSan Francisco, CA +1

Builds and scales internal data platform by designing data models, pipelines, and analytics infrastructure to transform raw product/business data into reliable datasets for company-wide decision-making. Partners with stakeholders across Product, Engineering, Finance, Marketing, and Sales.

180k – 250k/yr
HybridData Engineering
xAI

Software Engineer - Data Platform

xAIPalo Alto, CA +1

Builds and operates petabyte-scale data platform infrastructure using Kafka, Spark, Flink, and Trino to power real-time ML pipelines and analytics. Requires expertise in distributed systems, stream processing, and systems languages like Rust, Go, or Scala.

180k – 440k/yr
HybridData Engineering
Datology AI

Software Engineer, Data Infrastructure

Datology AIRedwood City, CA

Builds and maintains scalable data processing pipelines and backend systems for a data curation platform that optimizes training data for ML models. Partners with researchers to integrate research capabilities, ensuring reliability and security for customer data.

180k – 300k/yr
On-siteData Engineering
Airtable

Data Engineer

AirtableSan Francisco, CA +4

Designs and owns mission-critical data pipelines to enable decision-making across data science, growth, sales, marketing, and product teams. Requires 5+ years experience with scalable pipelines (preferably Airflow), Python, and advanced SQL.

180k – 222k/yr
On-site5+ YOEData Engineering