# Senior Software Engineer - Crawler

**Company:** [ZoomInfo](https://hotfix.jobs/companies/zoominfo)
**Location:** Remote
**Role:** Data Engineering
**Salary:** $140k – $220k/yr
**Experience:** 5+ years
**Skills:** Java, Python, GCP, AWS, Kubernetes, GKE, BigQuery, Snowflake, Apache Airflow, Apache Kafka, Terraform, Scrapy, ETL, Distributed Systems, Web Crawling
**Posted:** 2026-08-17

> Builds and operates large-scale web crawling, extraction, and data engineering infrastructure processing billions of pages. The role requires 5+ years of software engineering experience, strong distributed-systems fundamentals, and proficiency with Java or Python, cloud platforms, Kubernetes, and ETL technologies.

## Job Description

## Responsibilities
- Design and implement components of scalable, fault-tolerant web crawling and extraction pipelines.
- Write clean, production-grade code in Java and Python.
- Build and operate ETL/ELT pipelines for large-scale data extraction and transformation.
- Work with cloud infrastructure on GCP and AWS, primarily on GKE.
- Improve observability, reliability, and operational excellence across contributed systems.
- Partner with product and data science teams to deliver impactful solutions.
- Contribute to code reviews, documentation, and knowledge sharing.
- Stay current with web technologies, anti-crawling mechanisms, and AI-powered extraction approaches.

## Requirements
- 5+ years of professional software engineering experience building production systems.
- Strong computer science fundamentals, including algorithms, data structures, concurrency, and distributed systems.
- Proficiency in Java and/or Python.
- Experience owning features end-to-end, from design through deployment and operation.
- Ability to make sound component-level architectural decisions.
- Hands-on experience with cloud data warehouses such as BigQuery or Snowflake.
- Experience designing and operating large-scale ETL/ELT pipelines.
- Experience with orchestration tools such as Apache Airflow.
- Experience with streaming or event-driven systems such as Apache Kafka.
- Production experience on GCP or AWS; multi-cloud exposure is a plus.
- Hands-on experience with Kubernetes, including GKE or EKS, for distributed workloads.
- Familiarity with infrastructure-as-code tools such as Terraform.
- Strong communication, pragmatic problem-solving, and ability to operate in ambiguity.

## Nice-to-Haves
- Web crawling at scale using Scrapy or similar frameworks.
- Proxy infrastructure, rotation strategies, or anti-bot evasion techniques.
- Structured and unstructured web data extraction from diverse site architectures.
- SERP extraction.
- AI/LLM-based extraction approaches applied to HTML at scale.
- Experience in a B2B data company or data-as-a-product environment.

## Compensation and Benefits
- US base salary: **$140,000–$220,000 USD**.
- Additional compensation such as bonus, commission, equity, and other benefits may apply.
- Comprehensive benefits and holistic mind, body, and lifestyle programs.

## Similar jobs

- [Database Admin Engineer](https://hotfix.jobs/jobs/9be8f2cb-d14c-4f82-b4e5-e1bda45419d2) - Mark43 - Boston, MA - $140k – $180k/yr
- [Senior Software Engineer](https://hotfix.jobs/jobs/8438d41b-d083-42da-9e19-5f127a824fad) - ZoomInfo - Waltham, MA - $140k – $220k/yr
- [People Intelligence Architect](https://hotfix.jobs/jobs/7d4200df-d2c4-4e3f-a084-c627945db590) - Square - California - $143k – $258k/yr
- [Marketing Analytics Engineer](https://hotfix.jobs/jobs/5db57de9-17ac-4e98-a1b0-496221fc4e65) - NinjaTrader - Chicago, IL - $135k – $185k/yr
- [Senior Analytics Engineer II](https://hotfix.jobs/jobs/baabea06-6a36-493a-9b3d-a709083402d9) - Wrapbook - Remote - CA$148k – CA$203k/yr

**Apply:** https://hotfix.jobs/jobs/40301940-d8a3-41ce-8665-d65327d78fd8
**Canonical:** https://hotfix.jobs/jobs/40301940-d8a3-41ce-8665-d65327d78fd8