Staff Software Engineer, Foundations
Leads the re-platforming of Vanta’s compliance data layer from MongoDB to schema-aware PostgreSQL across high-throughput Kafka and S3 pipelines. The role requires staff-level distributed systems expertise, migration leadership, and strong experience with relational and document data modeling.
About the job
Responsibilities
- Lead migration of the resource data model from MongoDB to a schema-aware, PostgreSQL-backed architecture while preserving public API contracts and customer data.
- Drive cross-team technical solutions for a shared data platform.
- Design for correctness under eventual consistency using idempotent session handling, conditional writes, reconciliation, and explicit backpressure.
- Design and evolve high-throughput ingestion and pipeline architecture for terabyte-scale distributed data streams.
- Productionize the Query API with schema versioning, joins, exports, tenant isolation, and predictable latency.
- Build auditable data infrastructure for compliance evidence.
- Diagnose production issues including hot partitions, unbounded fan-out, and online rewrites of continuously written tables.
- Set architectural direction for Kafka, event queuing, Redis, PostgreSQL, and MongoDB.
- Mentor senior engineers through design reviews, architectural guidance, and hands-on contributions.
- Champion reliability, observability, and operational hygiene.
- Use AI tools responsibly to accelerate engineering work and improve systems.
Requirements
- Proven experience leading platform migrations while maintaining backward compatibility and availability.
- Strong command of Kafka or equivalent streaming and queuing infrastructure, including partitioning, consumer groups, redelivery, and idempotency.
- Deep experience with distributed systems and data pipeline engineering at enterprise scale.
- Fluency in relational and document data modeling at scale, including query planning, connection pooling, and online schema changes.
- Experience defining multi-quarter technical roadmaps and delivering them with engineering leadership.
- Solid AWS fundamentals and proficiency with TypeScript and Node.js.
- Ability to explain complex systems tradeoffs to technical and non-technical audiences.
- Curiosity and sound judgment in applying AI responsibly.
Compensation & Benefits
- Salary and equity.
- Comprehensive medical, dental, and vision coverage; employee-only premiums are fully covered for most medical plans.
- 16 weeks of paid parental leave.
- Health and wellness stipend.
- Remote workspace, internet, and cellphone stipends.
- Commuter benefits for employees reporting to the San Francisco and New York City offices.
- Family planning benefits.
- Matching 401(k) contribution with immediate vesting.
- Flexible paid time off and 80 hours of sick time.
- 11 company-paid holidays.
- Virtual team-building activities, lunch-and-learns, and company-wide events.
Skills
Kafka, Postgres, MongoDB, Redis, AWS, TypeScript, Node.js, S3, Distributed Systems, Data Pipelines, Eventual Consistency, Schema Versioning, Query Planning, Connection Pooling
Similar jobs
Data Engineering jobs
Vercel
San Francisco, CAStaff Data Platform Engineer leading the architecture and development of financial data infrastructure for revenue reporting, billing, forecasting, and compliance. Requires 8+ years of data engineering or architecture experience, strong streaming and warehouse expertise, and the ability to mentor engineers and partner with Finance and Audit leaders.
Anyscale
San Francisco, CA
Staff Software Engineer responsible for designing and scaling Ray Data’s distributed data-processing infrastructure for large-scale AI training and inference. Requires 6+ years of production software and architectural ownership experience, plus deep distributed-systems expertise and strong Python skills.
Haus
San Francisco, CAStaff-level engineer leading backend services and data-platform architecture, including large-scale ingestion, distributed systems, and trustworthy BigQuery/dbt warehouse models. Requires 10+ years of software engineering experience, expert Python, deep SQL/dbt expertise, and strong technical leadership.
Snowflake
Menlo Park, CA
Leads the design and operation of highly available distributed data platforms and pipelines at Snowflake, while providing technical leadership across teams. Requires 12+ years of distributed-systems experience, cloud expertise, and strong database and system-design depth.
Harvey
San Francisco, CAStaff Software Engineer on the central data platform team, responsible for architecture, ingestion, orchestration, streaming, governance, and self-service data tooling. Requires 10+ years building production data infrastructure and deep experience with warehouses, CDC, streaming, orchestration, Python, and SQL.