Software Engineer, Ray Data
Build and optimize Ray’s distributed data-processing and Datasets libraries, working across Ray Core, Apache Arrow, machine-learning integrations, and streaming workloads. The role requires at least five years of relevant experience and strong distributed-systems, algorithms, and data-processing expertise.
About the job
Responsibilities
- Develop high-quality open-source software to simplify distributed programming with Ray.
- Identify, implement, and evaluate architectural improvements to Ray Core and Ray Datasets.
- Build, optimize, and scale Ray’s Datasets library and data-processing capabilities.
- Improve Ray’s testing process to make releases as smooth as possible.
- Communicate work to a broader audience through talks, tutorials, and blog posts.
Projects
- Improve Ray Datasets performance at large scale using Apache Arrow primitives and Ray’s object manager.
- Integrate Ray with machine-learning training systems and data sources.
- Build stability and stress-testing infrastructure.
- Integrate streaming workloads, such as Beam on Ray.
- Differentiate data operations in Anyscale’s hosted Ray service.
Requirements
- At least 5 years of relevant work experience.
- Strong background in algorithms, data structures, and system design.
- Experience building scalable and fault-tolerant distributed systems.
- Experience with data processing and database internals.
- Experience with Spark or Dask; streaming experience is a plus.
Skills
Python, C++, Ray, Ray Datasets, Apache Arrow, Distributed Systems, Algorithms, Data Structures, System Design, Data Processing, Database Internals, Spark, Dask, Streaming
Similar jobs
Backend Engineering jobs
Anyscale
San Francisco, CADevelop and improve Ray Core’s C++ distributed-systems backend, focusing on performance, reliability, fault tolerance, and scalability. The role requires at least five years of experience with distributed systems, C/C++, low-level operating systems, algorithms, and system design.
Rain
Remote
Design and lead secure, scalable wallet infrastructure spanning custody, key management, signing, authorization, and recovery. The role requires deep expertise in wallet or cryptographic security infrastructure, distributed systems architecture, and modern blockchain account models.
GitLab
Bengaluru, India
Build and operate production backend features, APIs, integrations, and services while collaborating asynchronously across product and engineering teams. The role requires strong backend fundamentals, independent feature ownership, and practical experience with modern backend languages and AI coding tools.
GitLab
Bengaluru, India
Build and operate production backend features, APIs, integrations, and services for a large-scale DevSecOps platform. The role requires professional backend development experience, strong database and testing fundamentals, independent feature ownership, and clear asynchronous communication.
Supabase
Remote
Build and operate a globally distributed CI/CD execution platform, including branch orchestration, ephemeral build environments, scheduling, caching, and pipeline observability. Requires Go and TypeScript expertise plus at least 3 years building cloud infrastructure or distributed systems.