Software Engineer, Web Crawling
150k – 300kSan Francisco, CABackend EngineeringOnsite
Summary
Builds and scales Google-scale web crawlers to handle 100M+ pages daily, optimizing for politeness, dynamic content, anti-bot measures, and prioritization. Requires experience with high-performance languages like Rust/C++ and tools like TypeScript/Playwright.
About the role
Responsibilities
- Build a distributed crawler that can handle 100M+ pages per day
- Optimize crawl politeness and rate limiting across thousands of domains
- Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures
- Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency
Desired Experience
- Extensive experience building and scaling web crawlers, or excited to ramp up quickly
- Experience with high performance languages (C++, Rust, etc.)
- Familiarity with TypeScript, Playwright, modern web design, CDP (Chrome DevTools Protocol)
- Comfortable optimizing systems to an exceptional degree
- Care about finding high quality knowledge
Skills
RustC++TypeScriptPlaywrightChrome DevTools ProtocolDistributed SystemsWeb CrawlingJavaScript RenderingRate LimitingCrawl Scheduling
Similar roles at this salary range
All Backend Engineering jobs →Member of Technical Staff
Write and maintain production backend services, build scalable frameworks, and deploy infrastructure using Java, Python, React, Docker, and Kubernetes. Requires 1 year of experience with LLMs, recommendation systems, and probabilistic modeling.
180k – 220kPalo Alto, CABackend EngineeringOn-site1+ YOEJavaScala