# Software Engineer

**Company:** [Spellbrush](https://hotfix.jobs/companies/spellbrush)
**Location:** San Francisco, CA
**Role:** Data Engineering
**Experience:** 3+ years
**Skills:** Python, TypeScript, Ray, Spark, Airflow, AWS, GCP, Azure, Snowflake, Redshift, BigQuery, S3, SQL, Distributed Systems
**Posted:** 2024-10-30

> Builds scalable data pipelines and infrastructure for AI research, processing petabyte-scale anime data across 10k GPUs. Partners with researchers using distributed systems, big data tools, cloud services, requires 3+ years generalist experience.

## Job Description

## Responsibilities
- Define processes and infrastructure to transform and make data (embeddings, video, bounding boxes) readily available across the company.
- Partner with AI researchers to understand needs, design, build, and monitor scalable pipelines.

## Requirements
- 3+ years as software engineer with generalist skillset.
- Experience with distributed systems and big data tools (Ray, Spark, Airflow).
- Hands-on experience shipping scalable data solutions in cloud (AWS, GCP, Azure) across data stores (Snowflake, Redshift, Hive, SQL/NoSQL).
- Comfortable pulling data from various sources (complex SQL on BigQuery, transferring PB-scale data across S3 regions).
- Highly comfortable scripting in TypeScript, Python, or Bash.
- Know when to use complex tools vs. simple solutions.

## Nice-to-Haves
- Experience designing/building highly scalable/reliable data pipelines, especially for AI.
- Interest in interfacing with AI researchers and ingesting model embeddings.
- Comfortable working on small, fast-paced teams.

## Similar roles

- [Data Analytics](https://hotfix.jobs/jobs/b8dd4634-fd75-4c54-8f56-f1c6d2ca6b99) - OnePay - Remote - $130k – $170k/yr
- [Forward Deployed Data Engineer (Integration)](https://hotfix.jobs/jobs/2c2de93d-e128-430e-a6c1-c51b1b338a8a) - Hilbert - San Francisco, CA
- [Software Engineer, Data Infrastructure](https://hotfix.jobs/jobs/a8009f18-5785-47c4-a091-169beaee14fd) - The Voleon Group - Remote - $235k – $300k/yr
- [Client Delivery Specialist](https://hotfix.jobs/jobs/6fa1ca24-291c-4451-862b-4e20168ad45b) - Hinge Health - San Francisco, CA - $80k – $120k/yr
- [Software Engineer](https://hotfix.jobs/jobs/8bf697ae-8d3c-4693-b178-f1f9771d729a) - xAI - Palo Alto, CA - $125k – $400k/yr

**Apply:** https://hotfix.jobs/jobs/65b9c40b-dd11-4491-b6fe-99314db271d0
**Canonical:** https://hotfix.jobs/jobs/65b9c40b-dd11-4491-b6fe-99314db271d0