Senior Data Platform Engineer
Senior Data Platform Engineer owning data infrastructure for identity and fraud detection products. Build scalable ETL/ELT pipelines, data observability, and storage layers using Python/Golang, Spark, AWS, and databases. 5+ years experience required; mentor juniors and collaborate with product/DS teams.
About the job
Responsibilities
- Build, expand, and optimize data infrastructure to create the most accurate dataset of identities and their relationships.
- Develop and operate secure, scalable, and reliable data ingestion and ETL/ELT pipelines that meet product requirements.
- Design and maintain a data observability framework to ensure data meets strict quality and freshness standards.
- Optimize data storage layer and build/maintain interfaces for scalable and fast data access.
- Collaborate with product teams to support data platform needs for delivery of existing and new products.
- Participate in call rotation for production issues.
- Mentor junior engineers and contribute to engineering best practices.
- Drive innovation by participating in hackathons and proof of concepts.
- Develop functional subject matter expertise in identity fraud domain.
Requirements
- 5+ years of experience in software engineering, data engineering or related field.
- Proficient in Python or Golang and related technologies and frameworks.
- Expertise in building and maintaining ETL/ELT pipelines at scale using distributed data processing technologies like Spark, Hadoop, Kafka or similar.
- Hands-on experience with public cloud platforms such as AWS, Microsoft Azure or GCP.
- Deep understanding of different database technologies including RDBMS (e.g. Postgres), NoSQL (OpenSearch, vector DB), Columnar data stores etc., and experience with efficient queries and optimization.
- Experience building enterprise grade, scalable, containerized data services and frameworks on Kubernetes or similar platforms.
- Working knowledge of Infrastructure-as-Code and DevOps practices.
- Excellent analytical and problem solving skills, interpersonal skills.
- Self-organized with ability to work independently and with ambiguity.
- Experience working in a Scrum / Agile development environment.
Nice-to-Haves
- Experience working with Spark/EMR.
- Built streaming applications.
- Experience with AWS technologies such as EKS, SQS/SNS, EMR, Redshift, S3 etc.
- Prior experience working in a fintech startup.
Compensation
$170,000 - $230,000 per year + equity + benefits.
Skills
Python, Go, Postgres, Opensearch, Redshift, Emr, Spark, Docker, AWS Lambda, AWS, ETL, ELT, Kafka, Hadoop, Kubernetes
Similar jobs
Data Engineering jobs
Crusoe
San Francisco, CA
Build foundational data platform infrastructure for analytics, machine learning, and cloud operations. The role requires 5+ years of software or data infrastructure engineering experience, strong Python and systems programming skills, distributed systems knowledge, and advanced SQL expertise.
Vercel
San Francisco, CAOwns the Finance data infrastructure supporting billing, usage-based revenue, forecasting, reporting, and close. The role requires production data engineering experience, strong SQL and Python, dbt and orchestration expertise, Finance-domain fluency, and the ability to mentor engineers and partner with business stakeholders.
Bevi
Boston, MA
Own and evolve Bevi’s end-to-end data platform, from ingestion and IoT modeling through governed self-service analytics and AI access. The senior individual contributor will architect scalable streaming and batch systems, establish governance and observability, and provide technical leadership across the Data & Data Science organization.
Clay
New York, NYBuild customer-facing data products and shared platform systems that transform conflicting, constantly changing sources into reliable, searchable information. The role requires 8+ years of hands-on engineering experience, strong Python and SQL skills, and ownership of product quality, reliability, and delivery.
Rippling
San Francisco, CA
Senior Software Engineer building and operating broad data platform infrastructure for data, analytics, ML, AI, and agent workflows. Requires 5+ years of software engineering experience with distributed systems, production data platforms, cloud infrastructure, and systems design.