Staff Software Engineer, Batch and Realtime Streaming
Architect and build Voleon’s batch and realtime streaming platform that powers ML research and production trading systems. Requires 10+ years experience building scalable data infrastructure with Python, Go, and distributed systems.
About the job
Responsibilities
- Architect, design, and implement core components for a Batch and Realtime Streaming platform, including data ingestion pipelines, storage systems, serving layers, and API interfaces
- Ship new features by collaborating across research, legal, trading, finance operations data, and infra teams for trading systems
- Collaborate with ML researchers and data scientists to understand their workflows and design intuitive interfaces (APIs, SDKs, UIs) for seamless feature discovery, access, and reuse
- Ensure data quality, consistency, and lineage for features, building robust mechanisms for versioning, monitoring, and governance
- Optimize data pipelines and storage for high performance, scalability, and reliability, considering both batch and real-time use cases
- Drive adoption of the feature store across teams by producing documentation, onboarding materials, and developer support
- Mentor junior engineers and contribute to team best practices and technical excellence
Requirements
- Bachelor’s or Master’s degree in Computer Science, Engineering, or a related field (or equivalent experience)
- 10+ years of software engineering experience, with a strong background in distributed computing and data infrastructure
- Experience in programming languages such as Python and Go
- Understanding of database technologies (Postgres, MySQL, Cassandra, DuckDB) and experience with APIs (REST/gRPC)
- Strong problem-solving skills, with a focus on delivering high-quality, maintainable, and well-documented solutions
- Excellent communication and collaboration skills; ability to work closely with both engineering and research teams
Preferred Qualifications
- Experience building large-scale data pipelines and storage systems (e.g., Airflow, Spark, Ray, Iceberg, etc)
- Exposure to modern Python data science tooling (pandas, polars, dask, duckdb, etc)
- Experience with monitoring and observability tools for distributed systems (e.g., Prometheus, Grafana, ELK Stack)
- Prior experience working with feature stores (e.g., Feast, Hopsworks, or custom solutions)
Skills
Python, Go, Airflow, Spark, Ray, Iceberg, Postgres, MySQL, Cassandra, Duckdb, Rest, gRPC, pandas, Polars, Dask
Similar jobs
Data Engineering jobs
Anthropic
San Francisco, CABuild and operate the data foundations behind AI safeguards, including production pipelines, data stores, governance controls, and internal tooling. The role requires strong Python and SQL skills, production data-platform experience, and expertise in reliable, privacy-conscious systems across multiple cloud environments.
The Voleon Group
New York, NYLeads the design and delivery of scalable data infrastructure, services, and developer tooling while shaping technical direction and mentoring engineers. Requires 8+ years of software engineering experience, strong systems design expertise, and proficiency in a modern programming language.
Vercel
San Francisco, CAStaff Data Platform Engineer leading the architecture and development of financial data infrastructure for revenue reporting, billing, forecasting, and compliance. Requires 8+ years of data engineering or architecture experience, strong streaming and warehouse expertise, and the ability to mentor engineers and partner with Finance and Audit leaders.
Vanta
Remote
Leads the re-platforming of Vanta’s compliance data layer from MongoDB to schema-aware PostgreSQL across high-throughput Kafka and S3 pipelines. The role requires staff-level distributed systems expertise, migration leadership, and strong experience with relational and document data modeling.
Anyscale
San Francisco, CA
Staff Software Engineer responsible for designing and scaling Ray Data’s distributed data-processing infrastructure for large-scale AI training and inference. Requires 6+ years of production software and architectural ownership experience, plus deep distributed-systems expertise and strong Python skills.