Skip to content
Rad AIRad AISan Francisco, CA

Data Engineer

Senior Data Engineer building scalable data pipelines, infrastructure, and architecture on AWS using Spark, Metaflow, and orchestration tools. Requires 5+ years data engineering experience with big data technologies; ML/healthcare background is a plus.

145k – 190k
Hybrid5+ YOEData Engineering

About the role

What You’ll Be Doing

  • Design and implement the data architecture, ensuring scalability, flexibility, and efficiency using pipeline authoring tools like Metaflow and large-scale data processing technologies like Spark.
  • Define and extend our internal standards for style, maintenance, and best practices for a high-scale data platform.
  • Collaborate with researchers and other stakeholders to understand their data needs including model training and production monitoring systems and develop solutions that meet those requirements.
  • Take ownership of key data engineering projects and work independently to design, develop, and maintain high-quality data solutions.
  • Ensure data quality, integrity, and security by implementing robust data validation, monitoring, and access controls.
  • Evaluate and recommend data technologies and tools to improve the efficiency and effectiveness of the data engineering process.
  • Continuously monitor, maintain, and improve the performance and stability of the data infrastructure.

Who We’re Looking For

  • 5+ years relevant experience in data engineering.
  • Expertise in designing and developing distributed data pipelines using big data technologies on large scale data sets.
  • Deep and hands-on experience designing, planning, productionizing, maintaining and documenting reliable and scalable data infrastructure and data products in complex environments.
  • Solid experience with big data processing and analytics on AWS, using services such as Amazon EMR and AWS Batch.
  • Experience in large scale data processing technologies such as Spark.
  • Expertise in orchestrating workflows using tools like Metaflow.
  • Experience with various database technologies including SQL, NoSQL databases (e.g., AWS DynamoDB, ElasticSearch, Postgresql).
  • Hands-on experience with containerization technologies, such as Docker and Kubernetes.
  • Prior Software Engineering experience is a big plus.

Nice to Haves

  • Experience working at an early stage startup.
  • Experience in a HIPAA compliant environment.
  • Experience working on machine learning or healthcare related projects.

Skills

SparkMetaflowAWSEmrAws BatchDynamoDBElasticsearchPostgresDockerKubernetesSQLNoSQL

Similar roles

Data Engineering jobs
Scale AI

Field Engineer, Public Sector

Scale AISt. Louis, MO +1

Field Engineer building and deploying data pipelines, integrations, and backend systems for government customers at customer sites. Requires active Secret clearance, Python, ETL, cloud technologies, and >50% onsite availability.

140k – 290k
HybridData Engineering
Distyl AI

Research Engineer, Data

Distyl AISan Francisco, CA +1

Research Engineers at Distyl build data systems and pipelines that power reliable compound AI workflows in enterprise environments. They create data quality frameworks, synthetic data strategies, and evaluation tools while partnering with researchers and customers to turn raw data into production AI value.

150k – 250k
HybridData Engineering
Beacon Biosignals

Software Engineer III

Beacon BiosignalsBoston, MA

Software Engineer on the Datastore team building and scaling backend data infrastructure, event-driven pipelines, data models, and APIs for brain EEG data to support scientific and clinical analytics. Requires 5+ years backend experience with strong data engineering focus.

150k – 170k
Remote5+ YOEData Engineering
Machinify

Healthcare Data Analyst

MachinifyUnited States

Create advanced SQL/Spark SQL queries and prompt-engineered LLM workflows to transform healthcare claims data into clinical insights and automated policy tools. Requires 3-5 years SQL plus 2-3 years healthcare experience.

140k – 170k
Remote3+ YOEData Engineering
Confido Legal

Analytics Engineer

Confido LegalNew York, NY

Build and maintain Confido's centralized data warehouse and analytics infrastructure. Design scalable data models, establish data standards, and enable self-service analytics across the organization.

150k – 190k
On-siteData Engineering