# Software Engineer - Web Crawling

**Company:** [Tavily](https://hotfix.jobs/companies/tavily)
**Location:** Unspecified
**Role:** Backend Engineering
**Experience:** 3+ years
**Skills:** Python, Node.js, Go, C++, Playwright, Puppeteer, Chrome Devtools Protocol, Browser Internals, Network Protocols, Docker, Kubernetes, CI/CD, Artificial Intelligence, Retrieval-Augmented Generation, Distributed Systems
**Posted:** 2026-03-06

> Build distributed, production-grade systems that acquire and structure live web data for AI search and retrieval pipelines. The role requires 3–5 years of backend or systems engineering experience, strong Python or Node.js skills, and production browser automation expertise.

## Job Description

## Responsibilities
- Build distributed data acquisition systems that capture and structure the live web.
- Investigate and analyze browser internals, fingerprinting, and anti-automation systems to develop stealthy, adaptive orchestration layers.
- Prototype and deploy intelligent automation frameworks using Playwright, Puppeteer, and low-level browser control protocols (CDP).
- Conduct hands-on research into network flows, JavaScript-based protections, and emerging web standards affecting automation.
- Collaborate with AI and infrastructure teams to integrate real-time web data into retrieval pipelines and LLM-powered agents.
- Translate deep technical insights into production-grade components, balancing research freedom with engineering rigor.
- Continuously evolve capabilities in resilience, speed, and authenticity of web interaction.

## Requirements
- 3–5 years of experience as a backend or systems engineer, ideally working with large-scale, distributed, or web-facing infrastructure.
- Strong programming skills in Python or Node.js; experience with Go or C++ is a strong plus.
- Proven experience building and maintaining browser automation systems using Playwright, Puppeteer, or CDP in production environments.
- Solid understanding of browser internals, network protocols, and web interaction mechanisms.
- Experience designing high-performance, resilient systems that handle scale, concurrency, and complex orchestration.
- Strong debugging and analytical skills, including investigating edge cases, performance bottlenecks, and behavior under dynamic web conditions.
- Comfortable working in a fast-moving environment and collaborating with product, AI, and infrastructure teams.
- Excellent documentation and communication skills.

## Nice-to-haves
- Familiarity with cloud infrastructure, containerization, Docker, Kubernetes, and CI/CD.
- Experience integrating AI or retrieval systems into production pipelines.

## Compensation and Benefits
- The position is based in Israel and requires existing legal authorization to work in Israel.
- Visa sponsorship is not available for this role.

## Similar jobs

- [Software Engineer - Wallets](https://hotfix.jobs/jobs/32955ac3-4c1c-4a94-83a2-7e6f18965180) - Rain - Remote - $220k – $270k/yr
- [Software Engineer - Branching](https://hotfix.jobs/jobs/e74343fa-f4f1-4737-8d28-ddcadd6feb92) - Supabase - Remote
- [Intermediate Software Engineer, Security Factory: Vulnerability Management](https://hotfix.jobs/jobs/9f09cc9e-0cdb-435d-8308-8947f044ea04) - GitLab - Remote - $115k – $173k/yr
- [Senior Software Engineer - Edge Services](https://hotfix.jobs/jobs/da2f4ed8-5407-4159-84e0-b9a46bfd3528) - Viz.ai - Tel Aviv, Israel
- [Senior Software Engineer - Distributed Systems](https://hotfix.jobs/jobs/335fddce-18c1-495f-a579-6facebf7f046) - Datadog - Bordeaux, France

**Apply:** https://hotfix.jobs/jobs/1d08fb6b-a301-47ab-8e61-fe856540f0d5
**Canonical:** https://hotfix.jobs/jobs/1d08fb6b-a301-47ab-8e61-fe856540f0d5