Staff Software Engineer, Foundations
Leads the migration and evolution of Vanta’s multi-tenant data platform, designing highly reliable ingestion, storage, and query systems at terabyte scale. The role requires staff-level distributed-systems expertise, Kafka and database fluency, and the ability to drive architecture across teams.
About the job
Responsibilities
- Lead migration of Vanta’s resource data model from a Mongo-centric architecture to a schema-aware, Postgres-backed solution while maintaining backward compatibility and customer integrations.
- Drive cross-team technical solutions for platform dependencies.
- Design for correctness under eventual consistency, including idempotent session handling, conditional writes, reconciliation, and explicit backpressure.
- Design and evolve high-throughput data ingestion and pipeline architecture for terabyte-scale distributed data streams.
- Expand the Query API into a production platform with schema versioning, joins, exports, tenant isolation, and predictable latency.
- Build externally auditable data infrastructure for compliance evidence.
- Diagnose production-scale issues including hot partitions, unbounded fan-out, and online rewrites of continuously written tables.
- Set architectural direction for Kafka, event queuing, Redis, Postgres, and MongoDB.
- Mentor senior engineers through design reviews, architectural guidance, and hands-on contributions.
- Champion reliability, observability, and operational excellence.
- Use AI tools responsibly to improve engineering efficiency and impact.
Requirements
- Proven experience leading platform migrations while maintaining backward compatibility and availability.
- Strong command of Kafka or equivalent streaming and queuing infrastructure, including partitioning, consumer groups, redelivery, and idempotency.
- Deep experience with distributed systems and data pipeline engineering at enterprise scale.
- Experience designing and operating terabyte-scale ingestion and high-throughput event-processing systems.
- Fluency in relational and document data modeling at scale, particularly Postgres and MongoDB.
- Experience with query planning, connection pooling, and online schema changes on large continuously written tables.
- Experience defining multi-quarter technical roadmaps and delivering them with engineering leadership.
- Solid AWS fundamentals.
- Professional experience with TypeScript and Node.js.
- Ability to communicate complex systems tradeoffs to technical and non-technical audiences.
- Curiosity and sound judgment in applying AI to engineering work.
Compensation & Benefits
- Industry-competitive salary and equity.
- Medical, dental, and vision benefits with dependent coverage fully covered.
- Pension contribution.
- 16 weeks of paid parental leave for all new parents.
- Health and wellness stipend.
- Remote workspace, internet, and cellphone stipends.
- Flexible work hours and location.
- 21 days of vacation time and 80 hours of sick leave.
- 11 company-paid holidays.
- Virtual team-building activities, lunch and learns, and company-wide events.
Skills
Kafka, Postgres, MongoDB, Redis, AWS, TypeScript, Node.js, S3, Distributed Systems, Data Pipelines, Event Queuing, Schema Versioning, Query Planning, Connection Pooling, Online Schema Change
Similar jobs
Data Engineering jobsBuild and operate foundational streaming, messaging, and data pipeline infrastructure for highly scalable identity and analytics systems. The role requires 3+ years of software development experience and strengths in distributed systems, event streaming, and platform reliability.
Leads the design and improvement of large-scale data platform systems and workflows, collaborating across engineering, data science, and business teams. Requires 10+ years of relevant experience, advanced SQL and Python, cloud data tooling, and technical leadership.
Leads the re-platforming of Vanta’s compliance data layer from MongoDB to schema-aware PostgreSQL across high-throughput Kafka and S3 pipelines. The role requires staff-level distributed systems expertise, migration leadership, and strong experience with relational and document data modeling.
Staff engineer owning the analytical data layer, schema, and tiered analytics architecture. The role combines hands-on backend development with database performance optimization, observability, ingestion coordination, and measured architectural decision-making.
Build and operate scalable lakehouse infrastructure, streaming and CDC pipelines, query systems, and self-serve BI capabilities. Requires 5+ years of data engineering experience, strong Kubernetes and infrastructure-as-code expertise, and hands-on experience with distributed data platforms.