Senior Data Engineer
Designs and builds scalable batch/streaming data pipelines for identity verification products, owning end-to-end data initiatives using cloud-native tech. Requires 5+ years data engineering with Spark, AWS, Python/SQL; streaming/orchestration experience preferred.
About the job
What You'll Do
- Design and build batch and streaming data pipelines to support automated data ingestion, ML feature engineering and analytics across multiple product domains.
- Own end-to-end delivery of complex, ambiguous data initiatives, including architecture, implementation, testing, deployment, monitoring, and documentation.
- Develop and evolve the data platform to support large-scale data processing using modern cloud-native technologies.
- Automate data operations (validation, quality checks, alerting, backfills, and recovery workflows) to reduce manual effort and improve consistency.
- Optimize cost, performance, and reliability of data workloads.
- Partner closely with cross-functional teams (Data Science, Product, Engineering) to understand requirements, translate them into technical solutions.
- Evaluate and adopt new technologies (new processing engines, storage formats, orchestration tools, GenAI-assisted ingestion) to keep the platform modern and efficient.
What You Bring
- 5+ years of hands-on data engineering experience, building and maintaining production-grade data platforms and pipelines.
- Strong programming skills in general-purpose language (such as Python or Scala) for data processing, and SQL for data analytics.
- Deep experience with distributed data processing frameworks, such as Apache Spark, including performance tuning and optimization.
- Proven experience building data solutions using services on AWS (EMR, Lambda, S3, etc).
- Strong understanding of data modeling and data warehousing concepts, including partitioning, schema design for large-scale datasets.
- Experience operating and supporting production pipelines, including monitoring, alerting, incident response, and improving reliability over time.
- Solid foundation in software engineering practices, including version control, CI/CD, testing strategies, and code review.
- Strong communication and collaboration skills, with the ability to work effectively with both technical and non-technical stakeholders.
Preferred Qualifications
- Experience with streaming or near-real-time data processing (Kafka, Kinesis, etc).
- Hands-on experience with data orchestration tools (Airflow, Step Functions, etc).
- Familiarity with modern data platform patterns such as Data Lakehouse, Data Mesh, and large-scale data sharing across teams.
- Experience with prompt engineering using modern GenAI, Large Language Models (LLM).
- Experience mentoring other engineers and contributing to engineering-wide standards, best practices.
Skills
Python, Scala, SQL, Spark, AWS, Emr, AWS Lambda, S3, Kafka, Kinesis, Airflow, Step Functions, CI/CD, Generative AI, LLMs
Similar jobs
Data Engineering jobs
Underdog Fantasy
United States
Build and own the data platform powering regulatory reporting for a federally regulated prediction-markets business. The role combines dbt and SQL engineering, automated validation, root-cause investigations, audit support, and direct collaboration with regulators and compliance.
Granica
Mountain View, CA
Build and optimize foundational lakehouse infrastructure for AI, spanning metadata, transactions, table maintenance, storage layout, and query performance at massive scale. The role requires senior systems engineering experience with modern lakehouse technologies, columnar formats, cloud object storage, and systems-oriented programming languages.
Vanta
Remote
Senior Analytics Engineer responsible for designing complex data models, building scalable SQL pipelines, enabling AI tooling, and improving data infrastructure to support self-serve analytics, dashboards, and data science at Vanta. Requires 4+ years data experience, software engineering mindset, and expertise with modern analytics tools like dbt.
Deepgram
United States
Own the company’s metric governance program by defining canonical metrics, enforcing them in semantic and catalog systems, improving data quality, and validating AI-agent outputs. Requires 5+ years in analytics or analytics engineering, strong SQL, production semantic-layer ownership, and experience with AI evaluation and data governance.
Gusto
Denver, COThe Senior Data Engineer will build scalable pipelines, ETL workflows, and data products while partnering with analytics, product, and engineering teams. The role requires 8–10+ years of experience, strong SQL and programming skills, cloud data-platform expertise, and a focus on reliability, quality, and automation.