Senior Data Engineer, Core Experimentation
Build and manage data pipelines and canonical datasets for experimentation platform, tracking product metrics like user growth and revenue. Collaborate with cross-functional teams at OpenAI; requires 3+ years data engineering experience with Spark, ETL tools, and distributed systems.
About the job
Responsibilities
- Design, build and manage data pipelines, ensuring all user event data is seamlessly integrated into our data warehouse.
- Develop canonical datasets to track key product metrics including user growth, engagement, and revenue.
- Work collaboratively with various teams, including Infrastructure, Data Science, Product, Marketing, Finance, and Research to understand their data needs and provide solutions.
- Implement robust and fault-tolerant systems for data ingestion and processing.
- Participate in data architecture and engineering decisions, bringing your strong experience and knowledge to bear.
- Ensure the security, integrity, and compliance of data according to industry and company standards.
Requirements
- 3+ years of experience as a data engineer and 8+ years of any software engineering experience (including data engineering).
- Proficiency in at least one programming language commonly used within Data Engineering, such as Python, Scala, or Java.
- Experience with distributed processing technologies and frameworks, such as Hadoop, Flink and distributed storage systems (e.g., HDFS, S3).
- Expertise with any of ETL schedulers such as Airflow, Dagster, Prefect or similar frameworks.
- Solid understanding of Spark and ability to write, debug and optimize Spark code.
Compensation
$293K - $325K USD
Skills
Python, Scala, Java, Spark, Airflow, Dagster, Prefect, Hadoop, Flink, S3
Similar jobs
Data Engineering jobs
Zoox
Foster City, CA
Leads the architecture, scaling, security, governance, and cost optimization of enterprise and AI data platforms. Requires 10+ years of data or software engineering experience, with expertise in production data foundations, CI/CD, governance, security, and performance optimization.
Garner Health
New York, NY
Build and scale data pipelines, reusable datasets, and validation frameworks supporting business intelligence, marketing, and data science. The role requires strong Python and SQL skills, modern data-stack experience, and at least four years of software or data engineering experience.
Prompt Health
United States
Own the reliability, performance, observability, scalability, and cost efficiency of large Aurora MySQL production environments supporting healthcare applications. The role requires 6+ years of database engineering experience, deep MySQL and AWS expertise, and strong skills in automation, incident response, and query optimization.
OnePay
United States
Leads an analytics engineering team that transforms raw data into reliable, actionable insights for product, marketing, and operations. The role requires 7+ years in data or analytics engineering, management experience, and advanced SQL, Databricks, and dbt expertise.
Decagon
San Francisco, CASenior Data Infrastructure Engineer responsible for building and operating reliable, low-latency streaming and batch data systems that support AI products. Requires 5+ years of production data infrastructure experience and expertise with technologies such as Kafka, Flink, ClickHouse, and Terraform.