Member of Technical Staff
Build and operate large-scale web crawlers, ingestion pipelines, and distributed data systems supporting advanced search. The role requires at least five years of software development experience and expertise in one of Python, C++, Rust, or Go, plus Kubernetes and AWS.
Salary not listed
On-site5+ YOEFullstack Engineering
About the job
Responsibilities
- Lead projects focused on developing large-scale web crawlers, ingestion pipelines, and data processing systems.
- Build, maintain, and optimize backend and frontend components for crawler services, including storage, retrieval, and UI dashboards for data management.
- Collaborate with Search and Infrastructure teams to ensure reliable, high-quality ingestion and processing of billions of web pages.
- Architect and implement full-stack features and scalable distributed systems for high-load and real-time data operations.
- Iterate, experiment with novel approaches, and improve system performance, usability, and reliability.
Requirements
- At least 5 years of software development experience.
- Strong knowledge of data structures and algorithms in at least one of Python, C++, Rust, or Go.
- Experience building, deploying, and optimizing high-load, distributed, and hardware-adjacent services.
- Deep understanding of cloud infrastructure.
- Hands-on experience with Kubernetes and AWS.
- Ability to write clean, efficient, and scalable systems.
Nice to Have
- Experience with large-scale web crawlers.
Skills
PythonC++RustGoKubernetesAWSWeb CrawlersDistributed SystemsData StructuresAlgorithmsCloud InfrastructureReal-Time Systems