Engineering Manager - Pipelines Engine
Lead Pipelines Engine team building next-gen ETL features and Agentic Data Engineering infrastructure for Databricks Lakeflow platform. Requires 3+ years managing engineering teams and 5+ years in data infrastructure like Apache Spark.
About the job
Responsibilities
- Lead engineering team building next-generation ETL features for Databricks Lakeflow platform.
- Oversee recruitment of top-tier talent and upskilling team members.
- Build processes to implement product vision, strategy, organizational goals, and priorities.
- Build high-quality, easy-to-operate software.
- Manage technical debt, long-term technical architecture, and balance product roadmap.
Requirements
- Minimum 3 years managing top-tier engineering teams.
- 5+ years building data infrastructure systems (e.g., Apache Spark or database internals).
- Passion for database systems, storage systems, distributed systems, or performance optimization.
- Experience working with product management and customers to understand needs.
- Ensure team builds high-quality, reliable infrastructure; responsible for testing, quality, SLAs in complex domains like distributed data systems.
- Expertise in attracting, hiring, and coaching engineers to Databricks standards; up-leveling teams.
Skills
Spark, ETL, Distributed Systems, Data Infrastructure, Database Systems, Storage Systems, Performance Optimization, Lakeflow, Spark Declarative Pipelines, Genie Code
Similar jobs
Engineering Management jobs
TheGuarantors
New York, NY
Leads the engineering team responsible for property-management platform capabilities, data models, integrations, APIs, and production reliability. Requires 5+ years of software engineering experience, technical leadership, backend expertise, and strong database and distributed-systems knowledge.
1Password
United StatesLeads the engineering team responsible for launching and scaling Credential Governance, a product that helps enterprises discover, control, and manage company-owned credentials. Requires several years of engineering management experience, production-service leadership, technical fluency, and experience growing remote teams.
Datadog
New York, NY
Leads the Coordination Systems Storage team, managing five engineers while contributing to distributed-systems development, incident response, and reliability initiatives. Requires platform infrastructure experience, strong coding and performance skills, and a background in operational excellence.
Fivetran
Oakland, CALeads the Metadata engineering team responsible for discovery APIs, catalog, lineage, and run-history services. The manager owns delivery, architecture, reliability, operational health, stakeholder planning, and engineer development in a distributed cloud environment.
Rippling
Seattle, WALeads a technical engineering team building automated IT infrastructure, inventory systems, and device-lifecycle software. The role requires engineering management experience, strong distributed-systems and architecture expertise, and hands-on technical leadership in an ambiguous environment.