Senior Platform Engineer
Leads development of DataHub's ingestion framework, building scalable metadata systems, APIs, and event-driven architectures for enterprise AI and data platforms. Requires 4+ years in distributed systems and advanced Python expertise.
About the job
Responsibilities
- Build scalable, fault-tolerant ingestion systems for enterprise-scale metadata
- Develop clean, intuitive APIs for our connector ecosystem
- Create event-driven architectures for real-time metadata processing
- Implement schema mapping between diverse systems and DataHub's unified model
- Develop versioning systems for AI assets (training data, model weights, embeddings)
Requirements
- 4+ years building production-grade distributed systems
- Advanced Python expertise with a focus on API design
- Experience with high-scale data processing or integration frameworks
- Strong systems knowledge and distributed architecture experience
- A track record of solving complex technical challenges
Nice-to-Haves
- Experience with DataHub or similar metadata/ETL frameworks (Airflow, Airbyte, dbt)
- Open-source contributions
- Early-stage startup experience
Compensation
Salary Range: $225,000 to $300,000
Skills
Python, API Design, Distributed Systems, Data Processing Frameworks, Event-Driven Architecture, Schema Mapping, Airflow, Airbyte, dbt, Kubernetes
Similar jobs
DevOps / SRE jobs
The Voleon Group
Berkeley, CABuild and evolve the developer platform that enables reliable, efficient software delivery across the company. The role requires 5+ years of software engineering experience, strong programming and system-design fundamentals, and expertise in build systems, CI/CD, testing, and deployment automation.
Descript
San Francisco, CA
Own and evolve a broad infrastructure platform spanning cloud, Kubernetes, deployment, reliability, security, and GPU-backed AI systems. The role requires 8+ years operating production distributed systems, strong incident and architecture experience, and practical cloud infrastructure expertise.
Zoox
Foster City, CA
Senior engineer owning safety-critical software pipelines and infrastructure, from static and dynamic analysis through CI enforcement, dashboards, and reliability tooling. Requires an advanced technical degree, 7+ years working with large codebases, and expertise in Bazel, Python, backend infrastructure, and C++.
Vapi
San Francisco, CA
Own and improve the CI/CD, testing, and deployment infrastructure that enables fast, safe, observable releases at scale. The role requires strong distributed-systems expertise, hands-on Kubernetes and infrastructure-as-code experience, and a track record of measurable cross-team improvements.
Skydio
San Mateo, CA
Build and improve cloud infrastructure, developer workflows, and internal tooling that make software development, testing, and releases more efficient and reliable. The role requires cloud architecture knowledge, CI/CD experience, Terraform and Bazel proficiency, and software development skills in Go, Python, or C++.