Lead development of LLM observability features at Datadog, building tools for monitoring, tracing, and evaluating AI systems in production. Requires expertise in distributed systems, GenAI applications, and model internals.
234k – 300k/yr
HybridML Engineering
About the role
What You’ll Do
Drive design and implementation of LLM observability features
Ideate, prototype, and scale new product features for generative AI systems
Work cross-functionally with engineering, product, UX, and applied science teams
Develop tools for tracing, evaluating, and debugging LLMs
Influence architecture decisions and mentor engineers
Stay close to customer pain points to guide priorities
Stay current with ML and observability trends
Who You Are
BS/MS/PhD in Computer Science, Engineering, or related field (or equivalent)
Deep understanding of distributed systems and scalable backend architectures
Hands-on experience building and shipping LLM-powered or GenAI applications
Understanding of model internals, inference pipelines, evaluation techniques, and prompt engineering
Ability to thrive in ambiguous, fast-changing environments with product mindset
Excited to shape AI observability tools
Clear communicator with rigorous thinking and clean code
Technical leader on Datadog's APM team building and deploying GenAI/ML models for agentic investigations, automated troubleshooting, and incident triaging. Requires 10+ years experience leading large-scale GenAI initiatives end-to-end in product environments.
234k – 234k/yr
Hybrid10+ YOEML Engineering
Staff AI Engineer - Cortex Code Quality
SnowflakeMenlo Park, CA
Staff AI Engineer builds and owns quality systems for Cortex Code AI coding agents, including agent strategy, experimentation pipelines, failure analysis, and cross-team alignment. Requires 8+ years shipping AI/ML software, proficiency in Python/TypeScript/Go, and expertise in LLM evaluation harnesses.
236k – 339k/yr
On-site8+ YOEML Engineering
Staff Software Engineer, Agents
HarveySan Francisco, CA +1
Builds AI agent systems for legal workflows, optimizing performance via prompt engineering, model selection, tools, and evaluations. Partners with customers/PMs to ship low-latency agents using Python and LLM APIs; requires 8+ years experience.
231k – 340k/yr
Hybrid8+ YOEML Engineering
Senior Staff Software Engineer, AI Model LifeCycle
CrusoeSan Francisco, CA +1
Builds and manages platforms for AI model lifecycles, focusing on fine-tuning, training pipelines, and reinforcement learning for LLMs. Requires 8+ years in AI, advanced degree, and hands-on experience with generative AI techniques.
238k – 318k/yr
On-site8+ YOEML Engineering
Staff Machine Learning Engineer
RedditSan Francisco, CA
Staff Machine Learning Engineer designing, training, and deploying advanced ML models (DNNs, transformers, RL) for Reddit's large-scale online advertising ranking, optimization, and marketplace systems. Requires Master's, 3+ years experience, technical leadership, and expertise in production ML systems.