Skip to content
DatadogDatadogBoston, MA

Staff Software Engineer - ML Observability

Lead development of LLM observability features at Datadog, building tools for monitoring, tracing, and evaluating AI systems in production. Requires expertise in distributed systems, GenAI applications, and model internals.

234k – 300k/yr
HybridML Engineering

About the role

What You’ll Do

  • Drive design and implementation of LLM observability features
  • Ideate, prototype, and scale new product features for generative AI systems
  • Work cross-functionally with engineering, product, UX, and applied science teams
  • Develop tools for tracing, evaluating, and debugging LLMs
  • Influence architecture decisions and mentor engineers
  • Stay close to customer pain points to guide priorities
  • Stay current with ML and observability trends

Who You Are

  • BS/MS/PhD in Computer Science, Engineering, or related field (or equivalent)
  • Deep understanding of distributed systems and scalable backend architectures
  • Hands-on experience building and shipping LLM-powered or GenAI applications
  • Understanding of model internals, inference pipelines, evaluation techniques, and prompt engineering
  • Ability to thrive in ambiguous, fast-changing environments with product mindset
  • Excited to shape AI observability tools
  • Clear communicator with rigorous thinking and clean code
  • Experience with observability tools/platforms

Benefits and Growth

  • Build tools for software engineers
  • Influence product direction and business impact
  • Work with skilled teammates
  • Competitive global benefits
  • Continuous professional development

Skills

LLMsGenerative AIDistributed SystemsBackend ArchitecturesModel InferencePrompt EngineeringObservability ToolsLlm TracingModel EvaluationDrift Detection

Similar roles

ML Engineering jobs
Datadog

Staff GenAI Engineer - Application Performance Monitoring

DatadogNew York, NY

Technical leader on Datadog's APM team building and deploying GenAI/ML models for agentic investigations, automated troubleshooting, and incident triaging. Requires 10+ years experience leading large-scale GenAI initiatives end-to-end in product environments.

234k – 234k/yr
Hybrid10+ YOEML Engineering
Snowflake

Staff AI Engineer - Cortex Code Quality

SnowflakeMenlo Park, CA

Staff AI Engineer builds and owns quality systems for Cortex Code AI coding agents, including agent strategy, experimentation pipelines, failure analysis, and cross-team alignment. Requires 8+ years shipping AI/ML software, proficiency in Python/TypeScript/Go, and expertise in LLM evaluation harnesses.

236k – 339k/yr
On-site8+ YOEML Engineering
Harvey

Staff Software Engineer, Agents

HarveySan Francisco, CA +1

Builds AI agent systems for legal workflows, optimizing performance via prompt engineering, model selection, tools, and evaluations. Partners with customers/PMs to ship low-latency agents using Python and LLM APIs; requires 8+ years experience.

231k – 340k/yr
Hybrid8+ YOEML Engineering
Crusoe

Senior Staff Software Engineer, AI Model LifeCycle

CrusoeSan Francisco, CA +1

Builds and manages platforms for AI model lifecycles, focusing on fine-tuning, training pipelines, and reinforcement learning for LLMs. Requires 8+ years in AI, advanced degree, and hands-on experience with generative AI techniques.

238k – 318k/yr
On-site8+ YOEML Engineering
Reddit

Staff Machine Learning Engineer

RedditSan Francisco, CA

Staff Machine Learning Engineer designing, training, and deploying advanced ML models (DNNs, transformers, RL) for Reddit's large-scale online advertising ranking, optimization, and marketplace systems. Requires Master's, 3+ years experience, technical leadership, and expertise in production ML systems.

230k – 322k/yr
Remote7+ YOEML Engineering