# Senior Software Engineer, Data Platform

**Company:** [Parafin](https://hotfix.jobs/companies/parafin)
**Location:** San Francisco, CA
**Role:** Data Engineering
**Salary:** $230k – $265k/yr
**Experience:** 4+ years
**Skills:** Python, SQL, Pyspark, Databricks, Airflow, AWS, Terraform, Spark, Kafka, Delta Lake, dbt, Trino, Hadoop, Hive, Datadog
**Posted:** 2026-02-13

> Build and maintain scalable data pipelines and lakehouse infrastructure using PySpark, Databricks, and Airflow on AWS. Partner with Data Science and Engineering teams to enhance data quality, observability, and ML platform support. Requires 4+ years experience with Python, SQL, and cloud data stacks.

## Job Description

## What You’ll Do
- Design and build robust, highly scalable data pipelines and lakehouse infrastructure with **PySpark**, **Databricks**, and **Airflow** on **AWS**.
- Improve the data platform development experience for Engineering, Data Science, and Product by creating intuitive abstractions, self‑service tooling, and clear documentation.
- Own and maintain core data pipelines and models that power internal dashboards, ML models, and customer-facing products.
- Own the Data & ML platform infrastructure using **Terraform**, including end‑to‑end administration of **Databricks** workspaces: manage user access, monitor performance, optimize configurations (e.g., clusters, lakehouse settings), and ensure high availability of data pipelines.
- Lead projects to improve data quality, testing, observability, and cost efficiency across existing pipelines and backend systems (e.g., migrating **Databricks SQL** pipelines to **dbt**, scaling data ingestion, improving data-lineage tracking, and enhancing monitoring).
- Act as the primary engineering partner for the Data Science team—embedded closely to gather requirements, design scalable solutions, and provide end-to-end support on all engineering aspects of their work.
- Work closely with backend engineers and data scientists to design performant data models and support new product development initiatives.
- Share best practices and mentor other engineers working on data-centric systems.

## What We’re Looking For
- 4+ years of experience in software engineering with a strong background in data infrastructure, pipelines, and distributed systems.
- Advanced proficiency in **Python** and **SQL**.
- Hands-on **Spark** development experience.
- Expertise with modern cloud data stacks—**AWS** (S3, RDS), **Databricks**, and **Airflow**—and lakehouse architectures.
- Hands‑on experience with foundational data‑infrastructure technologies such as **Hadoop**, **Hive**, **Kafka** (or similar streaming platforms), **Delta Lake/Iceberg**, and distributed query engines like **Trino/Presto**.
- Familiarity with ingestion frameworks, developer‑experience tooling, and best practices for data versioning, lineage, partitioning, and clustering.
- Strong problem-solving skills and a proactive attitude toward ownership and platform health.
- Excellent communication and collaboration skills, especially in cross-functional settings.

**Bonus Points**
- Experience with **AWS** infrastructure using **Terraform**.
- Familiarity with observability tools (e.g., **Datadog**) and cost tracking in cloud environments.
- Experience with financial systems or building platforms in a fintech setting.
- Prior work on ML infrastructure: Feature stores (e.g., **Tecton**), ML model lifecycle (training, deployment, monitoring, retraining), real-time inference.
- Contributions to internal tooling or open-source projects in the data ecosystem.

## What We Offer
**Salary Range: $230k-$265k**
- Equity grant
- Medical, dental & vision insurance
- Work from home flexibility
- Unlimited PTO
- Commuter benefits
- Free lunches
- Paid parental leave
- 401(k)
- Employee assistance program

## Similar roles

- [Software Engineer, Habitat (Online Data)](https://hotfix.jobs/jobs/6d5f1e36-3a2d-4c87-a36c-c160e9794099) - OpenAI - Seattle, WA - $230k – $385k/yr
- [Senior Software Engineer, Post-Trade Financial Systems](https://hotfix.jobs/jobs/c7c51252-a63f-4e49-bfb6-0a1c3df6741f) - The Voleon Group - Berkeley, CA - $225k – $255k/yr
- [Lead Analytics Engineer](https://hotfix.jobs/jobs/9ded60e6-58ae-49a5-b38d-4e8b823e723b) - Zoox - Foster City, CA - $236k – $284k/yr
- [Senior Manager, Data Engineering](https://hotfix.jobs/jobs/ef6e5000-ec89-412e-82ba-1346f5480377) - Justworks - New York, NY - $240k – $310k/yr
- [Senior Data Engineer](https://hotfix.jobs/jobs/54f376ce-0c59-4864-bf87-05c654a7ec2d) - EliseAI - New York, NY - $240k – $300k/yr

**Apply:** https://hotfix.jobs/jobs/a1e663b4-ff6c-4f35-a3d1-85cf965f8ae9
**Canonical:** https://hotfix.jobs/jobs/a1e663b4-ff6c-4f35-a3d1-85cf965f8ae9