Data Engineer
Build scalable data pipelines, cloud-native infrastructure, and SQL intelligence systems for AI-powered data operations. The role requires 3–5 years of data engineering experience, strong Python and SQL skills, and expertise in query optimization, SQL parsing, and data pipelines.
About the job
Responsibilities
- Build highly performant, large-scale data infrastructure capable of supporting 100K+ jobs and petabyte-scale data volumes per day.
- Design and implement robust, scalable cloud-native data infrastructure on AWS.
- Use Kubernetes and Airflow for resource management and deployment.
- Develop a comprehensive SQL intelligence system for query optimization, dynamic pipeline generation, and data lineage tracking.
- Apply SQL query profiling, AST analysis, and parsing to improve query performance, build adaptive data pipelines, and implement granular column-level lineage.
- Integrate advanced AI capabilities into data systems and workflows.
- Contribute to open-source initiatives.
Requirements
- 3–5 years of experience in data engineering, focused on scalable data pipelines and systems.
- Strong proficiency in Python and SQL.
- Extensive experience with SQL query profiling, optimization, and performance tuning, preferably with Snowflake.
- Deep understanding of SQL Abstract Syntax Trees (ASTs) and experience with SQL parsers such as sqlglot.
- Experience generating column-level lineage and dynamic ETLs.
- Experience building data pipelines with Airflow or dbt.
Nice-to-haves
- Understanding of cloud platforms, particularly AWS.
- Familiarity with Kubernetes for containerized deployments.
Compensation and Benefits
- Opportunity to work on AI-powered data engineering and generative AI applications.
- High-visibility work supporting data teams worldwide.
- Open-source contribution opportunities.
- Career growth in AI-powered data engineering.
Skills
Python, SQL, Snowflake, Sql Ast, Sqlglot, Airflow, dbt, AWS, Kubernetes, Sql Optimization, Data Pipelines, Data Lineage
Similar jobs
Data Engineering jobs
Vanta
Remote
Own the systems that ingest, standardize, validate, and operationalize data signals for Vanta’s EPD organization. The role suits a hands-on builder who has recently shipped working tools or pipelines, uses AI-assisted development, and helps teammates grow technically.
Alpaca
Remote
Build and operate scalable lakehouse infrastructure, streaming and CDC pipelines, query systems, and self-serve BI capabilities. Requires 5+ years of data engineering experience, strong Kubernetes and infrastructure-as-code expertise, and hands-on experience with distributed data platforms.
StarTree
India
Build and operate distributed systems powering Apache Pinot’s real-time analytics platform at massive scale. The role requires strong distributed-systems expertise, end-to-end delivery ownership, and a focus on reliability, observability, and performance.
Earnin
Bengaluru, India
Senior data platform engineer who scales infrastructure, automates data delivery, builds AI-enabled analytical tools, and leads cross-functional engineering initiatives. Requires 4+ years of data infrastructure experience, strong Kafka and distributed-systems expertise, and proficiency in Python, Scala, cloud platforms, and Terraform.
Fivetran
Bengaluru, India
Senior Software Engineer building reliable connectors and high-volume data pipelines that move customer data into warehouses. The role requires strong Java, cloud, database, distributed-systems, and technical leadership experience.