Senior Software Engineer
Build and operate Databricks’ company-wide Data Intelligence Platform, including metrics stores, ETL frameworks, orchestration, governance, and reliable multi-cloud data pipelines. The role requires 6+ years of industry experience and technical leadership on large-scale data infrastructure projects.
About the job
Responsibilities
- Design and operate the metrics store, enabling business units and engineering teams to share and aggregate detailed metrics with high quality, introspection, and query performance.
- Design and operate a cross-company Data Intelligence Platform containing business and product metrics, balancing data protection with ease of sharing.
- Develop tooling and infrastructure to manage and operate Databricks at scale across multiple clouds, geographies, and deployment types.
- Build CI/CD processes, pipeline test frameworks, data-quality tooling, and infrastructure-as-code tooling.
- Design the base ETL framework used by company data pipelines.
- Partner with engineering teams to define the long-term vision and requirements for the Databricks product.
- Build reliable data pipelines and solve data problems using Databricks, partner products, and open-source tools.
- Provide feedback on the design and operation of data products.
- Establish conventions and create APIs for telemetry, debugging, feature, and audit-event data.
- Represent Databricks at academic and industry conferences and events.
Requirements
- 6+ years of industry experience.
- 4+ years providing technical leadership on large projects involving ETL frameworks, metrics stores, infrastructure management, or data security.
- Experience building, shipping, and operating reliable, multi-geography data pipelines at scale.
- Experience operating workflow or orchestration frameworks, including Airflow, dbt, or commercial enterprise tools.
- Experience with large-scale messaging systems such as Kafka, RabbitMQ, or commercial systems.
- Excellent cross-functional communication and consensus-building skills.
- Passion for data infrastructure and enabling others to access data more easily.
Benefits
- Comprehensive benefits and perks tailored to employees' regional needs.
Skills
Databricks, ETL, Data Pipelines, Metrics Stores, Apache Airflow, dbt, Kafka, RabbitMQ, CI/CD, Infrastructure As Code, Data Quality, Data Governance, Multi-Cloud, Python, Data Security
Similar jobs
Data Engineering jobs
Alpaca
Remote
Build and operate scalable lakehouse infrastructure, streaming and CDC pipelines, query systems, and self-serve BI capabilities. Requires 5+ years of data engineering experience, strong Kubernetes and infrastructure-as-code expertise, and hands-on experience with distributed data platforms.
StarTree
India
Build and operate distributed systems powering Apache Pinot’s real-time analytics platform at massive scale. The role requires strong distributed-systems expertise, end-to-end delivery ownership, and a focus on reliability, observability, and performance.
Earnin
Bengaluru, India
Senior data platform engineer who scales infrastructure, automates data delivery, builds AI-enabled analytical tools, and leads cross-functional engineering initiatives. Requires 4+ years of data infrastructure experience, strong Kafka and distributed-systems expertise, and proficiency in Python, Scala, cloud platforms, and Terraform.
Fivetran
Bengaluru, India
Senior Software Engineer building reliable connectors and high-volume data pipelines that move customer data into warehouses. The role requires strong Java, cloud, database, distributed-systems, and technical leadership experience.
Snowflake
Pune, India
Build and operate scalable enterprise data pipelines, models, and platform infrastructure across the full data lifecycle. The role requires 5+ years of experience, strong SQL and Python, and deep expertise in Snowflake, dbt, and Airflow.