Staff Software Engineer, Lakeflow Pipelines DR
Staff engineer designing distributed systems for cross-region replication, failover, and recovery of Lakeflow data pipelines. The role requires strong production software engineering skills and expertise in consistency, transactions, idempotency, replication, and related systems, with 8+ years of experience preferred.
About the job
Responsibilities
- Design and implement distributed systems that replicate and recover Lakeflow pipelines across regions.
- Build cross-region replication and recovery for pipelines, streaming tables, and materialized views.
- Develop consistency mechanisms across pipeline dependencies, table versions, and transaction logs.
- Implement failover and failback workflows with conservative correctness guardrails.
- Build deep clone, metadata reconciliation, observability, and failure-injection testing capabilities.
- Develop high-fidelity recovery simulations and game-day testing.
- Apply formal reasoning to failure modes and deliver incremental, production-quality systems toward a multi-year technical vision.
Requirements
- Strong software engineering skills in Java, Scala, C++, Go, Python, or a similar production language.
- Understanding of consistency, transactions, idempotency, replication, checkpointing, and data lineage.
- Passion for distributed systems, databases, storage systems, streaming systems, or reliability engineering.
- 8+ years of experience working on related systems preferred.
Nice to Have
- PhD or advanced research experience in databases, distributed systems, or storage.
Compensation
- Local pay range: $192,000–$260,000 USD.
- May include annual performance bonus, equity, and benefits.
Skills
Java, Scala, C++, Go, Python, Distributed Systems, Databases, Storage Systems, Streaming Systems, Reliability Engineering, Transactions, Replication, Checkpointing, Data Lineage
Similar jobs
Data Engineering jobs
Databricks
New York, NY
Leads large-scale advertising data ingestion, measurement, and agentic workflow systems, combining deep ad-tech expertise with production LLM experience. Requires 10+ years of engineering experience and technical and people leadership in complex enterprise environments.
Rippling
San Francisco, CABuild and scale data ingestion platforms, pipelines, APIs, and processing products that move billions of rows across a multi-tenant system. The role requires 8+ years of software development experience and strong expertise in large-scale application architecture.
Webflow
United States
Staff Data Engineer leading data platform initiatives across batch, streaming, real-time pipelines, data lake infrastructure, governance, and privacy. Requires 5+ years of data engineering experience, strong Spark and distributed processing expertise, and the ability to lead complex production systems.
IntusCare
United States
Staff-level engineer responsible for the technical direction, reliability, and evolution of a cloud ELT platform supporting healthcare data products. The role requires 7+ years of software or data engineering experience, deep SQL/Python and modern data-platform expertise, and strong architectural and mentoring leadership.
Rula
United States
Leads organization-wide Snowflake migration, Medallion architecture, warehouse optimization, and CI/CD quality controls while partnering with executives on data strategy. The role requires 6+ years in analytics or data engineering, expert SQL, production dbt experience, and strong architectural judgment.