Staff Software Engineer
Leads the architecture and performance of high-throughput real-time streaming and Lakehouse data platforms. The role requires 8+ years of software engineering experience, deep Scala or Java and JVM expertise, extensive Kafka experience, and strong AWS capabilities.
About the job
Responsibilities
- Architect distributed, high-throughput streaming architectures using Kafka Streams, Apache Spark, and Apache Flink.
- Build the roadmap for transitioning to a robust Lakehouse architecture.
- Identify and resolve system bottlenecks while maintaining pipeline stability, data integrity, and low latency at massive event volumes.
- Establish engineering standards for reliability, monitoring, observability, and automated testing.
- Use Datadog and Prometheus for monitoring and observability.
- Identify opportunities to refactor and modernize the platform for efficiency and cost-effectiveness.
Requirements
- 8+ years of software engineering experience.
- At least 5 years of production expertise in Scala or Java, including concurrency, memory management, and garbage-collection tuning.
- Deep JVM knowledge.
- At least 5 years of hands-on experience with the Kafka ecosystem.
- Expertise in Kafka offsets, partitions, rebalancing, and state stores.
- Experience with Apache Kafka, Kafka Streams, Kafka Connect, Apache Flink, and Apache Spark.
- Experience designing resilient systems under heavy load.
- Understanding of consistency-model and storage-format trade-offs.
- Mastery of AWS for handling terabytes of data, including MSK, EMR, Athena, and Lambda.
- Experience with Terraform and Kubernetes.
Compensation & Benefits
- Competitive salary.
- Insurance, annual leave, bonuses, referral rewards, and other benefits.
- Hybrid work model with 3 days in the office at Prestige Tech Pacific.
Interview Process
- Coding interview.
- System design discussion.
- Hiring manager meeting covering team, values, and culture.
- Leadership discussion focused on AI and solution design.
- Final discussion with HR or senior leadership.
Skills
Scala, Java, Jvm, Apache Kafka, Kafka Streams, Kafka Connect, Apache Flink, Spark, AWS, Terraform, Kubernetes, Datadog, Prometheus, Lakehouse, Aws Msk
Similar jobs
Data Engineering jobs
Celonis
Bengaluru, India
Leads the design and development of scalable analytic data infrastructure, including the foundation for Celonis’s Digital Twin. Requires 10+ years of analytics or data engineering experience, enterprise Databricks expertise, and strong stakeholder communication.
Vanta
Remote
Leads the re-platforming of Vanta’s compliance data layer from MongoDB to schema-aware PostgreSQL across high-throughput Kafka and S3 pipelines. The role requires staff-level distributed systems expertise, migration leadership, and strong experience with relational and document data modeling.
Databricks
Bengaluru, India
Leads the design and operation of Databricks’ large-scale Data Intelligence Platform, including metrics stores, ETL frameworks, multi-cloud pipelines, governance, and infrastructure tooling. Requires extensive industry experience, distributed-systems expertise, and technical leadership across complex data infrastructure initiatives.
Databricks
Bengaluru, India
Leads the design, operation, and evolution of Databricks’ cross-company Data Intelligence Platform, including large-scale data systems, pipelines, governance, and infrastructure. Requires 10+ years of distributed-systems experience and substantial technical leadership on production data platforms.
Alpaca
Remote
Build and operate scalable lakehouse infrastructure, streaming and CDC pipelines, query systems, and self-serve BI capabilities. Requires 5+ years of data engineering experience, strong Kubernetes and infrastructure-as-code expertise, and hands-on experience with distributed data platforms.