Research Engineer, QC Automation
Build QC automation systems for RL training data and agent evals at HUDHUD. Design quality standards, validation pipelines, experiments and metrics without heavy LLM reliance; partner with vendors to debug and improve data generation. Requires Python, Docker, Linux and experience building scalable QA/QC systems end-to-end.
About the job
Responsibilities
- Create QC systems based on true understanding and human judgement, without relying heavily on LLMs
- Define and enforce quality standards for training data
- Design experiments and metrics to grade agent outputs
- Partner with data vendors to debug quality issues and diagnose agent failure modes, provide actionable feedback, and improve their data generation processes
- Translate QC learnings into systems for auditing supplier-generated datasets, including sampling strategies, validation pipelines (rule-based and model-assisted), and feedback loops
- Continuously integrate QC learnings into infrastructure tools and data vendor portal to reduce anomalies, inconsistencies, and edge cases
Requirements
- Proficiency in Python, Docker, and Linux environments
- Strong understanding of what “good data” means and how to measure it
- Built scalable data validation pipelines and automated QA/QC systems end-to-end without a fully prescribed roadmap
- Experience working on benchmarks and evals - you can reason about what makes a task realistic, a rubric reliable, an environment usable, and a trajectory useful for RL training
- Early-stage startup experience with ability to work independently in fast-paced environments
Nice-to-Haves
- Detail-oriented and able to spot subtle inconsistencies or edge cases in data
- Comfortable designing metrics, experiments, and QA/QC processes, not just executing them
- Experience with existing benchmarks and can reason about how to construct tasks in new evals
- Thrive in unstructured problem spaces
- Strong communication skills for remote collaboration across time zones
We prioritize technical aptitude and learning potential over years of experience. Motivated candidates are encouraged to apply even if they don't meet all criteria.
Compensation & Benefits
- Competitive compensation based on experience and location
- 100% covered top-of-the-line medical, dental, and vision from Blue Shield of CA
- Lunch and dinner when you’re in the office
- Company-wide holiday break (Christmas Eve to New Year’s Day) on top of PTO and paid holidays
- Other perks including an Equinox membership, 401k, and commuter benefits
- Unlimited* access to tokens for ChatGPT, Claude Code, Cursor, etc.
Skills
Python, Docker, Linux, Data Validation, Qa/Qc Systems, Benchmarks, Evals, Metrics Design, Rl Training, Sampling Strategies, Validation Pipelines
Similar jobs
ML Engineering jobs
Thinking Machines Lab
San Francisco, CA
Build and operate the engineering systems that support post-training research, including reinforcement learning infrastructure, sandboxed execution, data pipelines, and agent scaffolding. The role requires strong Python and systems engineering skills, project ownership, and a relevant bachelor’s degree or equivalent experience.
OpenAI
San Francisco, CA
Build research infrastructure and tooling that enables AI models to design silicon, including reinforcement learning environments, EDA integrations, evaluations, and experiment workflows. The role requires strong software engineering fundamentals and comfort working across research, tooling, and chip-design systems.
Rollstack
United StatesBuild production AI capabilities for automated slide and document generation, working across LLM applications, data analysis, and content generation. The role requires 3+ years in machine learning and NLP, advanced Python, and experience with LLM frameworks and production systems.
ClickUp
United States
Build and operate large-scale ranking and retrieval systems that power search relevance, including hybrid lexical/vector search, embeddings, query understanding, and permission-aware retrieval. Requires a bachelor's degree and 5+ years of ML engineering experience in ranking or information retrieval.
PathAI
Boston, MADevelop and deploy machine learning models for biomedical research and AI products, collaborating with scientific, engineering, and product teams. Requires an advanced quantitative degree, substantial ML experience, Python proficiency, and experience bringing models into production or research applications.