Staff Software Engineer, Data Platform
Staff Software Engineer on the central data platform team, responsible for architecture, ingestion, orchestration, streaming, governance, and self-service data tooling. Requires 10+ years building production data infrastructure and deep experience with warehouses, CDC, streaming, orchestration, Python, and SQL.
About the job
Responsibilities
- Own the data platform architecture and technical direction, including reusable frameworks and build-versus-buy decisions.
- Build and operate ingestion across streaming, batch, CDC, and third-party connectors.
- Design schema evolution that safely absorbs upstream changes.
- Land data in Snowflake with predictable freshness, completeness, and cost characteristics.
- Own orchestration for scheduling, retries, backfills, and dependency management.
- Build transformation, compute, and self-service pipeline frameworks.
- Design and operate stream-processing infrastructure for real-time product features, alerting, and reporting.
- Build data quality, observability, reconciliation, anomaly detection, lineage, cataloging, and discovery capabilities.
- Develop tooling for PII classification, masking, retention, access control, and multi-region data residency.
- Establish technical standards through design reviews, documentation, and mentorship.
Requirements
- 10+ years building and operating production data infrastructure and systems used by other teams.
- Deep experience with cloud data warehouses, especially Snowflake, including performance tuning and cost management.
- Experience building CDC and streaming pipelines with Kafka, Debezium, Flink, or Spark Streaming.
- Experience with managed ingestion tools such as Fivetran or Airbyte.
- Strong fluency with workflow orchestration tools such as Temporal, Airflow, or Dagster.
- Strong Python programming and advanced SQL skills.
- Experience building frameworks or internal tooling for other engineers.
- Experience with data quality, observability, lineage, and schema evolution.
- Working knowledge of data governance in regulated environments, including PII classification, masking, access control, retention, and data residency.
- Familiarity with Azure, AWS, or GCP, Kubernetes, and infrastructure-as-code tools such as Terraform or Pulumi.
- Ability to operate in ambiguity and define scope.
Nice to Have
- Experience with dbt and analytics engineering teams.
- Experience with lakehouse architectures, Iceberg, Delta Lake, or Trino.
- Experience operating multi-tenant platforms with strict security, compliance, or data residency requirements.
- Exposure to data infrastructure for AI products.
- Experience as an early or founding data platform hire at a fast-growing company.
Compensation
- $231,000–$340,000 USD annual compensation.
Skills
Snowflake, Python, SQL, Kafka, Debezium, Apache Flink, Spark, Fivetran, Airbyte, Airflow, Kubernetes, Terraform, dbt, Apache Iceberg, Delta Lake
Similar jobs
Data Engineering jobs
Snowflake
Menlo Park, CA
Leads the design and operation of highly available distributed data platforms and pipelines at Snowflake, while providing technical leadership across teams. Requires 12+ years of distributed-systems experience, cloud expertise, and strong database and system-design depth.
Anyscale
San Francisco, CA
Staff Software Engineer responsible for designing and scaling Ray Data’s distributed data-processing infrastructure for large-scale AI training and inference. Requires 6+ years of production software and architectural ownership experience, plus deep distributed-systems expertise and strong Python skills.
Haus
San Francisco, CAStaff-level engineer leading backend services and data-platform architecture, including large-scale ingestion, distributed systems, and trustworthy BigQuery/dbt warehouse models. Requires 10+ years of software engineering experience, expert Python, deep SQL/dbt expertise, and strong technical leadership.
OpenAI
San Francisco, CAAnalytics Engineer supporting Go-to-Market teams by building scalable data models, metrics, pipelines, visualizations, and self-service products. The role requires 10+ years of data experience, deep SQL expertise, Python proficiency, and strong business judgment.
Airbnb
United States
Staff Software Engineer leading development of data catalog and metadata infrastructure for discovery, governance, lineage, and quality across Airbnb’s data ecosystem. Requires 9+ years of software engineering experience focused on data infrastructure and strong programming and distributed data technology skills.