Skip to content
character.aicharacter.aiRedwood City, CA

Principal Research Engineer, Post-Training

Lead technical vision and execution for post-training systems that transform foundation models into intelligent, engaging products. Drive alignment algorithms, RL, and infrastructure for large-scale LLM training and serving.

275k – 400k/yr
On-site7+ YOEML Engineering

About the role

What You'll Do

Technical Leadership & Mentorship

  • Define and drive the technical roadmap for mid- and post-training systems, balancing research innovation with production reliability and scalability
  • Mentor and grow a team of researchers and engineers through technical guidance, design reviews, and career development
  • Establish best practices for experimentation, model development, and deployment

Research & Model Development

  • Lead the development of alignment algorithms, optimization techniques, and training objectives to improve model capabilities and data efficiency
  • Drive advances in mid- and post-training methodologies including reinforcement learning, preference optimization, supervised fine-tuning, and emerging alignment approaches
  • Identify and execute high-impact research opportunities that improve model behavior, safety, and user engagement
  • Develop robust evaluation frameworks and quality signals to measure real-world model performance

Systems & Infrastructure

  • Lead the design of efficient training and inference systems for large-scale generative models
  • Architect scalable data pipelines that transform diverse data sources into high-quality training datasets
  • Partner with infrastructure teams to optimize distributed training, GPU utilization, and serving efficiency
  • Drive improvements in experimentation platforms, data quality systems, and model observability

Required Qualifications

  • PhD in Computer Science, Machine Learning, AI, or a related field, or equivalent industry experience
  • Significant experience leading technical projects or teams in machine learning, AI research, or large-scale distributed systems
  • Experience scaling and mentoring high-performing research and engineering teams
  • Deep understanding of modern machine learning techniques, including transformers, reinforcement learning, alignment methods, and large language models
  • Strong track record of delivering impactful research or applied ML systems in production environments
  • Expertise in designing, building, and maintaining production-quality ML systems and infrastructure
  • Experience training, serving, debugging, and optimizing large-scale models on GPU-based systems
  • Experience leading teams working on large language model training, mid-training, or post-training
  • Experience with product experimentation, online evaluation, and A/B testing frameworks
  • Strong software engineering skills with the ability to write clean, maintainable, and scalable code
  • Excellent communication skills and the ability to influence technical direction across teams
  • Lead complex, cross-functional initiatives across data, training infrastructure, evaluation, and model serving

Nice to Have

  • Hands-on experience working directly with open-source models like Mistral and Qwen, particularly adapting them via mid- and post-training for specific personas, creative writing, or role-playing applications
  • Familiarity with cloud-native ML infrastructure, including Kubernetes, Docker, and modern orchestration platforms
  • Publications in leading machine learning conferences or demonstrated contributions to the broader AI community

Skills

TransformersReinforcement LearningLLMsAlignment MethodsGpu-Based SystemsDistributed TrainingKubernetesDockerPythonA/B Testing

Similar roles

ML Engineering jobs
Shield AI

Principal Engineer, State Estimation

Shield AIDallas, TX

Principal State Estimation Engineer developing advanced sensor fusion, inertial navigation, and real-time state estimation algorithms for autonomous UAVs in GPS-denied environments. Requires 12+ years experience with Kalman filters, C++, and shipping navigation products.

275k – 412k/yr
On-site12+ YOEML Engineering
Rula

Principal AI Engineer (Remote)

RulaLos Angeles, CA

Owns AI foundation for generative AI in mental healthcare, architecting production-grade systems like RAG and agentic loops while ensuring technical excellence and safety in regulated environments. Requires 10+ years software engineering, AI/ML expertise, and cloud/distributed systems experience.

281k – 330k/yr
Remote10+ YOEML Engineering
Airbnb

Principal Machine Learning Engineer- LLM Fine-tuning and Optimization

AirbnbUnited States

Principal MLE focused on fine-tuning and optimizing large language models for production, building scalable AI pipelines, and shaping Airbnb's ML platform strategy. Requires 10+ years experience and a PhD.

292k – 365k/yr
Remote10+ YOEML Engineering
Fivetran

Sr. Principal AI Systems Architect

FivetranOakland, CA

Designs and owns architecture for production-grade AI analyst systems, including prompts, tools, context, evaluation, and safety. Partners with teams to build reliable AI capabilities for analytical workflows, emphasizing scalability and quality.

252k – 315k/yr
HybridML Engineering
Redpanda Data

Principal Software Engineer, AI Platform

Redpanda DataSan Francisco, CA

Principal engineer architecting and building Redpanda's Agentic Data Plane, enabling AI agents to safely interact with enterprise data via streaming, query, and governance layers. Requires 10+ years building large-scale distributed systems and experience with agentic AI infrastructure.

245k – 290k/yr
Hybrid10+ YOEML Engineering