Sr. Staff Software Engineer, Big Data Platform
Leads the strategy, architecture, and development of large-scale data infrastructure powering big data and AI applications. Requires 10+ years of industry experience, deep expertise in Kubernetes and big data technologies, and proficiency in a programming language.
About the job
Responsibilities
- Lead the strategy and technical direction of Pinterest’s data infrastructure for big data and AI applications.
- Build and scale infrastructure frameworks for petabyte-scale datasets, including compute engines, job management, resource management, scheduling, and remote shuffling.
- Partner with internal customers on critical business use cases that rely on big data.
- Provide company-wide technical leadership on reliable, fast, and efficient data processing and storage at scale.
- Contribute to the team’s technical vision and long-term roadmap.
Requirements
- 10+ years of industry experience with a proven track record of technical excellence.
- 5+ years building and supporting large-scale Kubernetes or big data platforms.
- Deep knowledge of big data and machine learning technologies, such as Flink, Spark, Presto, Kubernetes, Ray, PyTorch, or TensorFlow.
- Proficiency in one or more programming languages, including Java, Go, Scala, or Python.
- Experience with Kubernetes and AWS technologies.
- Exceptional collaboration skills, including navigating ambiguity, making tradeoffs, and aligning stakeholders on priorities and progress.
- Bachelor’s degree in Computer Science, a related technical field, or equivalent experience.
Skills
Apache Flink, Spark, Presto, Kubernetes, Ray, PyTorch, TensorFlow, Java, Go, Scala, Python, AWS, Big Data, Machine Learning
Similar jobs
Data Engineering jobs
SmithRx
United States
Leads enterprise data engineering strategy, architecture, delivery, governance, and technical leadership across the organization. Requires extensive data engineering experience, advanced data modeling and warehouse expertise, and strong PySpark, SQL, and Python skills.
Airbnb
United States
Staff Software Engineer leading development of data catalog and metadata infrastructure for discovery, governance, lineage, and quality across Airbnb’s data ecosystem. Requires 9+ years of software engineering experience focused on data infrastructure and strong programming and distributed data technology skills.
Anyscale
San Francisco, CA
Staff Software Engineer responsible for designing and scaling Ray Data’s distributed data-processing infrastructure for large-scale AI training and inference. Requires 6+ years of production software and architectural ownership experience, plus deep distributed-systems expertise and strong Python skills.
United States
Leads the design, operation, and technical direction of Pinterest’s data workflow and context control planes, driving reliability, scalability, AI-native capabilities, and open-source contributions. Requires 10+ years of distributed-systems experience, infrastructure expertise, and proficiency in Python or Java.
Mozilla
United StatesLeads the design and improvement of large-scale data platform systems and workflows, collaborating across engineering, data science, and business teams. Requires 10+ years of relevant experience, advanced SQL and Python, cloud data tooling, and technical leadership.