Software Engineer, Web Crawling
Builds and scales Google-scale web crawlers to handle 100M+ pages daily, optimizing for politeness, dynamic content, anti-bot measures, and prioritization. Requires experience with high-performance languages like Rust/C++ and tools like TypeScript/Playwright.
About the job
Responsibilities
- Build a distributed crawler that can handle 100M+ pages per day
- Optimize crawl politeness and rate limiting across thousands of domains
- Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures
- Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency
Desired Experience
- Extensive experience building and scaling web crawlers, or excited to ramp up quickly
- Experience with high performance languages (C++, Rust, etc.)
- Familiarity with TypeScript, Playwright, modern web design, CDP (Chrome DevTools Protocol)
- Comfortable optimizing systems to an exceptional degree
- Care about finding high quality knowledge
Skills
Rust, C++, TypeScript, Playwright, Chrome Devtools Protocol, Distributed Systems, Web Crawling, Javascript Rendering, Rate Limiting, Crawl Scheduling
Similar jobs
Backend Engineering jobs
Postman
Cupertino, CA
Build secure, scalable identity and access management platforms, APIs, and internal tooling using JavaScript, Node.js, React.js, and distributed-systems expertise. The role requires 5+ years of backend or distributed-systems experience and familiarity with authentication protocols is a plus.
Cribl
United States
Build backend systems and APIs that ingest, process, and route large-scale telemetry data in real time. The role requires strong computer science fundamentals, Node.js/TypeScript experience, distributed-systems knowledge, and ownership of reliable cloud software.
Censys
United StatesBuild and maintain highly available, scalable distributed systems, data pipelines, messaging systems, and services powering Internet intelligence products. The role requires 5+ years of distributed-systems experience and proficiency with Go, cloud platforms, databases, and messaging technologies.
Square
New York, NY
Build and maintain backend systems powering Cash App Taxes, translating complex federal and state tax requirements into reliable filing and e-file functionality. The role requires 4+ years of software development experience, strong system design skills, and a bachelor's degree or equivalent experience.
Figma
San Francisco, CABuild shared Ruby backend foundations for APIs, asynchronous jobs, workflow orchestration, and developer tooling. The role requires 5+ years of software engineering experience, including 3+ years with production Ruby systems, plus expertise in distributed systems and large codebases.