# Software Engineer, Data Acquisition

**Company:** [OpenAI](https://hotfix.jobs/companies/openai)
**Location:** San Francisco, CA
**Role:** Data Engineering
**Salary:** $293k – $385k/yr
**Experience:** 4+ years
**Skills:** Kubernetes, Distributed Systems, Web Crawling, Data Processing, Infrastructure As Code, Key-Value Databases, Data Indexing, Search Algorithms, Backend Services, Data Ingestion
**Posted:** 2023-09-22

> Builds and leads data acquisition systems including web crawling, ingestion, and scalable distributed processing for model training. Requires 4+ years experience, expertise in Kubernetes and large-scale data systems, and BS/MS/PhD in Computer Science.

## Job Description

## Responsibilities
- Own and lead engineering projects in the area of data acquisition including web crawling, data ingestion, and search.
- Collaborate with other sub-teams, such as Data Processing, Architecture, and Scaling, to ensure smooth data flow and system operability.
- Work closely with the legal team to handle any compliance or data privacy-related matters.
- Develop and deploy highly scalable distributed systems capable of handling petabytes of data.
- Architect and implement algorithms for data indexing and search capabilities.
- Build and maintain backend services for data storage, including work with key-value databases and synchronization.
- Deploy solutions in a **Kubernetes** Infrastructure-as-Code environment and perform routine system checks.
- Conduct and analyze experiments on data to provide insights into system performance.

## Qualifications
- **BS/MS/PhD** in Computer Science or a related field.
- **4+ years** of industry experience in software development.
- Experience with large web crawlers a plus.
- Strong expertise in large stateful distributed systems and data processing.
- Proficiency in **Kubernetes**, and Infrastructure-as-Code concepts.
- Willingness and enthusiasm for trying new approaches and technologies.
- Ability to handle multiple tasks and adapt to changing priorities.
- Strong communication skills, both written and verbal.

## Similar roles

- [Data Engineer, Scaling Analytics](https://hotfix.jobs/jobs/95f29a65-412b-4f98-ac4e-c97c6315ac70) - OpenAI - San Francisco, CA - $293k – $385k/yr
- [IT Controls Data Engineer](https://hotfix.jobs/jobs/d6fe5097-4a1e-473c-b90e-829663dfcd39) - OpenAI - San Francisco, CA - $293k – $385k/yr
- [Data Engineer, People Innovation Labs](https://hotfix.jobs/jobs/f1e5db56-aece-4405-b44b-0a59543295a6) - OpenAI - San Francisco, CA - $293k – $325k/yr
- [Research Engineer, Economic Research](https://hotfix.jobs/jobs/5e0524fe-291d-4682-914d-4a2040886ed6) - Anthropic - San Francisco, CA - $300k – $405k/yr
- [Analytics Data Engineer](https://hotfix.jobs/jobs/14ba0ea4-c99e-4bcf-a437-4a5cd0d2d09f) - Anthropic - San Francisco, CA - $275k – $370k/yr

**Apply:** https://hotfix.jobs/jobs/6279860a-f2d9-4951-9851-227bbba7cf89
**Canonical:** https://hotfix.jobs/jobs/6279860a-f2d9-4951-9851-227bbba7cf89