# Software Engineer, Web Crawling

**Company:** [Exa](https://hotfix.jobs/companies/exa)
**Location:** San Francisco, CA
**Role:** Backend Engineering
**Salary:** $150k – $300k/yr
**Skills:** Rust, C++, TypeScript, Playwright, Chrome Devtools Protocol, Distributed Systems, Web Crawling, Javascript Rendering, Rate Limiting, Crawl Scheduling
**Posted:** 2025-07-23

> Builds and scales Google-scale web crawlers to handle 100M+ pages daily, optimizing for politeness, dynamic content, anti-bot measures, and prioritization. Requires experience with high-performance languages like Rust/C++ and tools like TypeScript/Playwright.

## Job Description

## Responsibilities
- Build a distributed crawler that can handle 100M+ pages per day
- Optimize crawl politeness and rate limiting across thousands of domains
- Design systems to detect and handle dynamic content, JavaScript rendering, and anti-bot measures
- Create intelligent crawl scheduling and prioritization algorithms for maximum coverage efficiency

## Desired Experience
- Extensive experience building and scaling web crawlers, or excited to ramp up quickly
- Experience with high performance languages (**C++**, **Rust**, etc.)
- Familiarity with **TypeScript**, **Playwright**, modern web design, **CDP (Chrome DevTools Protocol)**
- Comfortable optimizing systems to an exceptional degree
- Care about finding high quality knowledge

## Similar roles

- [Software Engineer, Workers Deploy & Config](https://hotfix.jobs/jobs/373e862c-92c6-4938-907d-a09a407f2835) - Cloudflare - New York, NY - $150k – $206k/yr
- [Backend Software Engineer](https://hotfix.jobs/jobs/08928e42-7bb9-4744-9124-acc9cde7ae0e) - Doxel - San Francisco, CA - $150k – $200k/yr
- [Deliverability Engineer, Core Sending](https://hotfix.jobs/jobs/0d8528d0-1960-4a62-867c-c7a9d2b9e09a) - Resend - Remote - $150k – $180k/yr
- [Backend Engineer , Core Sending](https://hotfix.jobs/jobs/158a2892-6b94-4186-ae37-b322d0659fca) - Resend - Remote - $150k – $170k/yr
- [Software Engineer — Integrations](https://hotfix.jobs/jobs/13c07401-ac9b-4d48-8c31-8fc220a67151) - Kodex - Remote - $150k – $180k/yr

**Apply:** https://hotfix.jobs/jobs/0cb6264d-0c90-45fa-bbec-c38454127490
**Canonical:** https://hotfix.jobs/jobs/0cb6264d-0c90-45fa-bbec-c38454127490