# Senior Research Engineer, Post-training & Evaluation

**Company:** [Reddit](https://hotfix.jobs/companies/reddit)
**Location:** Remote
**Role:** ML Engineering
**Salary:** $230k – $322k/yr
**Experience:** 6+ years
**Skills:** Python, PyTorch, Hugging Face Transformers, vLLM, Lm-Eval-Harness, Fsdp2, Deepspeed Zero-3, Sft, Cpt, RLHF
**Posted:** 2026-06-12

> Own evaluation science and post-training methodology for Reddit's foundational LLMs. Define benchmarks, design model-as-a-judge systems, and set SFT recipes that turn base models into safe, Reddit-native endpoints.

## Job Description

## Responsibilities
- Define the "Reddit Benchmark" evaluation standard: Own the methodology for rigorously measuring model quality across Safety, Reasoning, representation/retrieval, and Reddit-specific knowledge.
- Own evaluation reliability and statistical rigor: Establish the science behind trustworthy evals — judge variance, multi-sample scoring, inter-rater/inter-sample agreement, sampling and temperature effects, and calibration of automated judges.
- Design model-as-a-judge methodology: Own judge selection, prompt design, calibration, and reliability for automated evaluation using frontier external models.
- Set post-training recipes and strategy: Design SFT recipes (data mixtures, curriculum, ablation strategy) that convert base models into helpful, well-aligned endpoints.
- Evaluate base and CPT checkpoints: Design checkpoint-selection methodology across CPT experiments and LR studies.
- Drive synthetic data generation strategy: Define and curate high-quality instruction and evaluation sets to improve generalization where human data is scarce.
- Partner with Safety Engineering: Translate high-level safety policy into concrete classification metrics, probe sets, and CI/CD unit tests.
- Diagnose post-training instability: Dive into loss curves and eval logs to identify alignment tax and capability degradation.
- Lead research direction: Set technical direction for evaluation and post-training across the team, mentor engineers and scientists.

## Requirements
- 6+ years of professional ML experience (or PhD + 4+) with a direct focus on LLM post-training and evaluation.
- PhD or MS in CS, ML, NLP, IR, or a related quantitative field — or equivalent industry research experience.
- Deep expertise in evaluation reliability: judge/sample variance, multi-sample scoring, calibration, statistical significance, and the failure modes of automated evaluation.
- Strong experience building custom, domain-specific evaluation harnesses (e.g., lm-eval-harness, Inspect AI, LightEval).
- Experience evaluating both generation and representation/classification: model-as-a-judge for generative quality and precision/recall, PR-AUC, retrieval/MTEB-style metrics, gold-label denoising, and label-noise handling.
- Deep understanding of Continuous Pre-training (CPT), Instruction Tuning (SFT), and how data quality shapes model behavior.
- Fluency in Python; strong data-pipeline and eval-harness engineering (e.g., Hugging Face Transformers, vLLM, lm-eval-harness). Working knowledge of PyTorch and distributed training (FSDP2, DeepSpeed ZeRO-3).

## Nice to Have
- Experience with MLflow or similar experiment-tracking frameworks.
- Familiarity with modern fine-tuning frameworks (Axolotl, TorchTune) and PyTorch-native training stacks (TorchTitan).
- Synthetic data generation techniques (e.g., Self-Instruct).
- Experience with preference optimization (DPO, RLHF, RLAIF, GRPO).
- Publications in NLP/ML/FAccT or related venues, or other evidence of research leadership.
- Experience evaluating multimodal models (embeddings, hateful-memes-style classification).

## Similar roles

- [Senior AI Engineer, Agentic Data Enrichment](https://hotfix.jobs/jobs/b8c6a9c2-b17f-4be4-af78-3f9ac0b761d6) - Baselayer - San Francisco, CA - $230k – $340k/yr
- [Technical Lead Manager](https://hotfix.jobs/jobs/459dd304-2f00-4641-bc79-7f32e42e51c1) - Instabase - San Francisco, CA - $230k – $270k/yr
- [Senior Machine Learning Engineer](https://hotfix.jobs/jobs/29ce42de-a923-433c-a975-ea7af29ca814) - Otter - Mountain View, CA - $230k – $265k/yr
- [Senior Research Engineer - Safety Tooling and Data](https://hotfix.jobs/jobs/ca7e0d7f-a90f-4333-8437-612d086669af) - Cohere - Remote - $230k – $380k/yr
- [Senior Software Engineer, ML Platform](https://hotfix.jobs/jobs/82b0c648-5a10-4c0d-becd-614789dd70e2) - Parafin - San Francisco, CA - $230k – $265k/yr

**Apply:** https://hotfix.jobs/jobs/70f97c9a-6e68-4cd2-a237-bde0a8ac9caa
**Canonical:** https://hotfix.jobs/jobs/70f97c9a-6e68-4cd2-a237-bde0a8ac9caa