Skip to content
xAIxAIPalo Alto, CA

Software Engineer - Data Platform

Builds and operates petabyte-scale data platform infrastructure using Kafka, Spark, Flink, and Trino to power real-time ML pipelines and analytics. Requires expertise in distributed systems, stream processing, and systems languages like Rust, Go, or Scala.

180k – 440k/yr
HybridData Engineering

About the role

About the Team

The Data Platform team builds and operates infrastructure for large-scale data transport and processing, managing Apache Kafka, HDFS, Spark, Flink, and Trino for real-time ML pipelines, feed ranking, experimentation, analytics, and observability at petabyte scale.

About the Role

Design, build, and operate distributed systems powering data movement and compute, processing trillions of events daily for scalability, performance, and reliability in product and ML workloads.

What You Will Do

  • Design and implement high-throughput, low-latency data ingestion and transport systems.
  • Scale and optimize multi-tenant Kafka infrastructure supporting real-time workloads.
  • Extend and tune Spark, Flink, and Trino for demanding production pipelines.
  • Build interfaces, APIs, and pipelines enabling teams to query, process, and move data at petabyte scale.
  • Debug and optimize distributed systems, with a focus on reliability and performance under load.
  • Collaborate with ML, product, and infrastructure teams to unblock critical data workflows.

Ideal Candidate

  • Proven expertise in distributed systems, stream processing, or large-scale data platforms.
  • Proficiency in Rust, Go, Scala or similar systems languages.
  • Hands-on experience with Kafka, Flink, Spark, Trino, or Hadoop in production.
  • Strong debugging, profiling, and performance optimization skills.
  • Track record of shipping and maintaining critical infrastructure.
  • Comfortable working in fast-moving, high-stakes environments with minimal guardrails.

Compensation and Benefits

Annual Salary Range: $180,000 - $440,000 USD
Base salary plus equity, comprehensive medical, vision, dental, 401(k), disability insurance, life insurance, and perks.

Skills

KafkaSparkFlinkTrinoHdfsRustGoScalaDistributed SystemsStream Processing

Similar roles

Data Engineering jobs
Actively AI

Data Engineer

Actively AINew York, NY

Own data and analytics end-to-end: architect internal systems, build metrics/dashboards, and translate customer and product signals into structured inputs for AI agents.

180k – 210k/yr
On-siteData Engineering
Baseten

Data Engineer

BasetenSan Francisco, CA +1

Builds and scales internal data platform by designing data models, pipelines, and analytics infrastructure to transform raw product/business data into reliable datasets for company-wide decision-making. Partners with stakeholders across Product, Engineering, Finance, Marketing, and Sales.

180k – 250k/yr
HybridData Engineering
Exa

Software Engineer, Distributed Data Systems

ExaSan Francisco, CA

Architects and builds massive-scale data infrastructure for web crawling, embedding model training, and real-time search, handling hundreds of petabytes. Requires expertise in lakehouse architectures, distributed processing pipelines, and streaming systems like Kafka and Flink.

180k – 350k/yr
On-siteData Engineering
Datology AI

Software Engineer, Data Infrastructure

Datology AIRedwood City, CA

Builds and maintains scalable data processing pipelines and backend systems for a data curation platform that optimizes training data for ML models. Partners with researchers to integrate research capabilities, ensuring reliability and security for customer data.

180k – 300k/yr
On-siteData Engineering
Airtable

Data Engineer

AirtableSan Francisco, CA +4

Designs and owns mission-critical data pipelines to enable decision-making across data science, growth, sales, marketing, and product teams. Requires 5+ years experience with scalable pipelines (preferably Airflow), Python, and advanced SQL.

180k – 222k/yr
On-site5+ YOEData Engineering