Member of Technical Staff
Build and operate large-scale web crawlers, ingestion pipelines, and distributed data systems supporting advanced search. The role requires at least five years of software development experience and expertise in one of Python, C++, Rust, or Go, plus Kubernetes and AWS.
About the job
Responsibilities
- Lead projects focused on developing large-scale web crawlers, ingestion pipelines, and data processing systems.
- Build, maintain, and optimize backend and frontend components for crawler services, including storage, retrieval, and UI dashboards for data management.
- Collaborate with Search and Infrastructure teams to ensure reliable, high-quality ingestion and processing of billions of web pages.
- Architect and implement full-stack features and scalable distributed systems for high-load and real-time data operations.
- Iterate, experiment with novel approaches, and improve system performance, usability, and reliability.
Requirements
- At least 5 years of software development experience.
- Strong knowledge of data structures and algorithms in at least one of Python, C++, Rust, or Go.
- Experience building, deploying, and optimizing high-load, distributed, and hardware-adjacent services.
- Deep understanding of cloud infrastructure.
- Hands-on experience with Kubernetes and AWS.
- Ability to write clean, efficient, and scalable systems.
Nice to Have
- Experience with large-scale web crawlers.
Skills
Python, C++, Rust, Go, Kubernetes, AWS, Web Crawlers, Distributed Systems, Data Structures, Algorithms, Cloud Infrastructure, Real-Time Systems
Similar jobs
Fullstack Engineering jobs
Improbable
Remote
Product Engineer responsible for building Bolter’s AI agent runtime, workspace, generated applications, and production reliability systems. The role requires strong full-stack TypeScript experience, product judgment, rapid prototyping ability, and depth in AI systems, real-time infrastructure, or developer tooling.
ElevenLabs
United KingdomBuild and scale full-stack Creative and Studio products spanning APIs, user interfaces, infrastructure, and generative AI workflows. The role emphasizes product ownership, solving complex engineering problems, and expertise in Python, TypeScript, and React; no formal degree or experience requirement is specified.
Dataiku
FranceBuild and enhance Dataiku DSS core features across data preparation, AI/ML, visualization, MLOps, platform, and governance. The role requires significant product-focused software engineering experience and a strong interest in combining scalable backend systems with exceptional frontend experiences.
Build
New York, NYBuild and own the user-facing platform for an agentic AI company, including workflow creation tools, client and internal interfaces, external access gateways, and organizational automation. The role requires strong product judgment, rapid execution, and end-to-end ownership.
Nerdio
United KingdomLeads a corporate development team building AI agents, automation, full-stack internal applications, and data capabilities. The role combines hands-on architecture and development with technical leadership, Azure deployment, CI/CD ownership, coaching, and internal evangelism.