Senior Software Engineer
Build and operate large-scale data acquisition pipelines, distributed processing systems, and production Java services across batch and streaming workloads. The role requires 5+ years of backend or data engineering experience, strong distributed-systems expertise, and proficiency with cloud data technologies.
About the job
Responsibilities
- Design, build, and operate large-scale data acquisition pipelines that ingest, validate, transform, enrich, and store high-volume raw data.
- Architect resilient ETL/ELT workflows for batch, streaming, scheduled, and event-driven processing.
- Develop production Java services and data-processing applications for ingestion, orchestration, enrichment, deduplication, and delivery.
- Build and improve systems using Apache Airflow, Apache Beam, Spark, Google Dataflow, DataProc, Kafka, and Pub/Sub.
- Define approaches for schema evolution, data contracts, data validation, backfills, replayability, and idempotent processing.
- Improve reliability, performance, scalability, and cost efficiency across pipelines and services.
- Implement observability, monitoring, and alerting for pipeline health, throughput, latency, failures, and data quality.
- Partner with product, data science, platform, and data quality teams to deliver production-ready systems.
- Contribute to technical designs, implementation plans, and modernization efforts.
Requirements
- 5+ years of professional software engineering experience focused on backend systems, data engineering, or distributed processing.
- Experience building and operating production data pipelines at scale.
- Deep proficiency with Java and object-oriented design.
- Hands-on expertise with Apache Beam, Apache Airflow, Spark, Google Dataflow, or DataProc.
- Strong experience with streaming systems such as Apache Kafka or Google Pub/Sub.
- Understanding of batch and streaming processing, data modeling, schema evolution, and data quality management.
- Experience designing ETL/ELT workflows for large structured and semi-structured datasets.
- Experience designing high-throughput, fault-tolerant backend services and distributed systems.
- Understanding of APIs, integration patterns, retries, backpressure, idempotency, and operational failure modes.
- Experience with large-scale storage and query technologies such as BigQuery, Snowflake, or Trino.
- Experience with a cloud provider, preferably Google Cloud.
- Experience operating production services with monitoring, logging, alerting, SLIs, and incident investigation.
- Bachelor's degree in Computer Science, Software Engineering, or a related field.
- Ability to translate business and data requirements into technical designs and implementation plans.
Nice to Have
- Kubernetes experience, especially GKE or EKS.
- Terraform or other infrastructure-as-code experience.
- Experience with Snowflake, BigQuery, Starburst/Trino, or similar query engines.
- Knowledge of integrations involving CRM systems, email/calendar APIs, third-party feeds, change data capture, or external data providers.
- Experience in a B2B data company, data marketplace, or data-as-a-product environment.
- Expert-level experience with Apache Spark or another distributed processing framework.
Compensation and Benefits
- Base salary: $140,000–$220,000 USD.
- Additional compensation such as bonus, commission, equity, and benefits may apply.
- Comprehensive benefits and mind, body, and lifestyle well-being programs.
Skills
Java, Apache Airflow, Apache Beam, Spark, Google Dataflow, Dataproc, Apache Kafka, Google Pub/Sub, BigQuery, Snowflake, Trino, GCP, Kubernetes, Terraform, Etl/Elt
Similar jobs
Data Engineering jobs
Mark43
Boston, MA
Own and scale Mark43’s production database infrastructure, focusing on MySQL administration, monitoring, performance, reliability, upgrades, and recovery. The role requires at least seven years of production database experience, cloud infrastructure expertise, and familiarity with Terraform.
ZoomInfo
United States
Builds and operates large-scale web crawling, extraction, and data engineering infrastructure processing billions of pages. The role requires 5+ years of software engineering experience, strong distributed-systems fundamentals, and proficiency with Java or Python, cloud platforms, Kubernetes, and ETL technologies.
Square
California
Senior individual contributor who architects and builds end-to-end people data systems, predictive models, and AI-agent workflows. Requires 8+ years of experience across data engineering and data science, with expertise in Python, SQL, machine learning, sensitive HR data, and agentic AI.
NinjaTrader
Chicago, IL
Senior individual contributor responsible for architecting shared dbt models, marketing attribution, data quality, and AI-driven analytics workflows. Requires 6+ years in analytics engineering or data, deep dbt and SQL expertise, modern data-stack experience, and strong marketing measurement knowledge.
Wrapbook
United StatesBuild and own Wrapbook’s analytics layer, including production pipelines, governed data models, canonical datasets, self-serve analytics, and monitoring. The role requires strong SQL and Python, modern warehouse experience, and 4+ years in data or analytics engineering.