Senior Research Data Engineer

What You’ll Do

• Own the gold data layer: Transform messy silver tables into curated, semantically rich, clean, and documented gold datasets suitable for AI model development, including reusable datasets and features. Maintain as products and needs evolve.
• Reverse-engineer data semantics: Collaborate with product engineers, clinical, and workflow experts to understand product usage and data creation. Analyze SQL queries, stored procedures, and technical definitions to capture provenance, semantics, clinical event sequencing, cross-module record linkage, and quirks.
• Bridge semantics with AI needs: Design and build gold data products with evolving documentation to support efficient AI-first foundations for model R&D.
• Curate datasets across modalities: For generative AI, RAG, predictive models, and other techniques, provide chunked/tagged unstructured content with rich metadata, point-in-time features, and clean labels. For classical ML, deliver model-ready tables.
• Build pipelines for reuse: Develop scheduled, observable transformations from silver to gold in Databricks/Spark. Enable researcher iteration on new features and data mixes without full rebuilds.
• Automate quality, filtering, and synthesis: Implement programmatic labeling, weak supervision, near-duplicate detection, boilerplate/noise removal, and LLM-API-driven synthetic data generation.
• Version and hand off: Maintain reproducible dataset snapshots with clean lineage and semantic definitions for downstream AI R&D reuse.

Required Skills and Experience

• 5+ years building production data systems, with at least 2 supporting ML or AI workloads.
• Track record of quickly learning complex new data domains via source code, expert interviews, and building reusable artifacts.
• Advanced Python, SQL, and PySpark/Databricks for large, messy data. Expert SQL for reverse-engineering business logic from complex stored procedures and queries.
• Deep Databricks ecosystem knowledge: Delta Lake, Unity Catalog, Spark/PySpark tuning, MLflow.
• AI domain literacy: Understanding of embeddings, tokenization, feature engineering, point-in-time correctness, train/validation/test splits, data drift, and differing data needs for classical ML vs. generative models.
• Data wrangling across modalities: Transform unstructured content (text, PDFs, transcripts, logs) and structured tabular data into clean, model-ready forms.
• Experience with AI-friendly data formats (Parquet, Hugging Face datasets) and storage optimizations (partitioning, sharding, caching) in Azure, AWS, or similar.
• Data quality, filtering, and synthesis pipelines: Programmatic labeling/weak supervision (e.g., Snorkel), near-duplicate detection (MinHash/LSH), content/quality filters, LLM-API synthetic data generation.
• Pipeline orchestration (Airflow, Databricks Workflows, Dagster, Prefect) and dataset versioning (Unity Catalog, feature stores).
• Experience with regulated/sensitive data (HIPAA or equivalent) and de-identification concepts.
• Git-based version control and CI/CD for data/code.
• Strong written documentation skills; ability to elicit requirements and tacit knowledge from technical and non-technical experts.
• Bachelor’s degree in computer science, data science, engineering, statistics, or related field (equivalent experience considered).

Preferred

• Hands-on EHR data experience in skilled nursing, long-term care, post-acute care, or senior living.
• Knowledge of clinical terminologies (ICD-10, SNOMED CT, LOINC) and standards (HL7v2, FHIR, CCDA).
• dbt for transformation and testing.
• Familiarity with ML frameworks (e.g., PyTorch) to debug data bottlenecks; experience with LLM/foundation model training or fine-tuning data pipelines.
• Clinical NLP, OCR, document parsing, or ASR/transcript pipeline experience.
• Data lineage and catalog tools.
• Prior embedding in an AI or ML research team.
• Master’s degree in relevant quantitative or computer science field.