Lead Data Platform Engineer - Enterprise, Data & AI
Leads the architecture, scaling, security, governance, and cost optimization of enterprise and AI data platforms. Requires 10+ years of data or software engineering experience, with expertise in production data foundations, CI/CD, governance, security, and performance optimization.
About the job
Responsibilities
- Architect, optimize, scale, and own an enterprise and AI data platform and high-throughput data fabric.
- Build scalable data infrastructure and autonomous, self-healing pipelines that handle schema evolution, detect anomalies, execute circuit breakers, and recover from failures.
- Define and enforce data governance, access controls, automated data-quality testing, schema-evolution policies, and metadata management.
- Implement storage lifecycle strategies, resource throttling, and query optimization for high-throughput, low-latency analytics and AI workloads.
- Optimize Databricks and AWS EMR environments for cost and performance.
- Build CI/CD deployment templates, environment isolation, testing frameworks, version-control standards, rollback mechanisms, and automated deployment testing.
- Establish security standards, RBAC, IAM roles, fine-grained data masking, audit controls, and data-retention policies.
- Develop observability and alerting for data platforms.
Requirements
- 10+ years of hands-on experience in data platform engineering or software engineering.
- Proven experience architecting and scaling production-grade data foundations.
- Deep expertise in performance and query tuning, compute-resource allocation, and compute-storage lifecycle policies.
- Experience with enterprise security, RBAC, Active Directory/IAM roles, data masking, compliance controls, audit trails, and data retention.
- Experience establishing data governance frameworks, schema-evolution controls, data-quality audits, and near-real-time observability.
- Hands-on experience building CI/CD pipelines, environment isolation, version control, rollback mechanisms, and automated deployment testing for data assets.
Nice-to-Haves
- Experience incorporating LLMs or generative AI into pipeline operations, including code generation, triage, root-cause analysis, reconciliation, backfills, or anomaly detection.
- Infrastructure-as-code experience with Terraform.
- Containerization and orchestration experience with EKS.
- GPU/CPU performance optimization and utilization experience.
- Experience in autonomous vehicles, robotics, or high-tech manufacturing.
Skills
Data Platforms, Data Engineering, Databricks, Aws Emr, CI/CD, RBAC, IAM, Active Directory, Data Governance, Data Quality, Terraform, Amazon Eks, LLMs, Generative AI, Query Optimization
Similar jobs
Data Engineering jobs
Garner Health
New York, NY
Build and scale data pipelines, reusable datasets, and validation frameworks supporting business intelligence, marketing, and data science. The role requires strong Python and SQL skills, modern data-stack experience, and at least four years of software or data engineering experience.
Prompt Health
United States
Own the reliability, performance, observability, scalability, and cost efficiency of large Aurora MySQL production environments supporting healthcare applications. The role requires 6+ years of database engineering experience, deep MySQL and AWS expertise, and strong skills in automation, incident response, and query optimization.
OnePay
United States
Leads an analytics engineering team that transforms raw data into reliable, actionable insights for product, marketing, and operations. The role requires 7+ years in data or analytics engineering, management experience, and advanced SQL, Databricks, and dbt expertise.
Decagon
San Francisco, CASenior Data Infrastructure Engineer responsible for building and operating reliable, low-latency streaming and batch data systems that support AI products. Requires 5+ years of production data infrastructure experience and expertise with technologies such as Kafka, Flink, ClickHouse, and Terraform.
Discord
United States
Build and operate petabyte-scale data infrastructure powering Discord’s insights and products. The role requires 5+ years of software engineering experience, strong programming skills, and experience with large-scale pipelines, streaming, orchestration, or data warehousing.