# Staff Software Engineer - ML Observability

**Company:** [Datadog](https://hotfix.jobs/companies/datadog)
**Location:** Boston, MA, New York, NY
**Role:** ML Engineering
**Salary:** $234k – $300k/yr
**Skills:** LLMs, Generative AI, Distributed Systems, Backend Architectures, Model Inference, Prompt Engineering, Observability Tools, Llm Tracing, Model Evaluation, Drift Detection
**Posted:** 2026-01-21

> Lead development of LLM observability features at Datadog, building tools for monitoring, tracing, and evaluating AI systems in production. Requires expertise in distributed systems, GenAI applications, and model internals.

## Job Description

## What You’ll Do
- Drive design and implementation of LLM observability features
- Ideate, prototype, and scale new product features for generative AI systems
- Work cross-functionally with engineering, product, UX, and applied science teams
- Develop tools for tracing, evaluating, and debugging LLMs
- Influence architecture decisions and mentor engineers
- Stay close to customer pain points to guide priorities
- Stay current with ML and observability trends

## Who You Are
- BS/MS/PhD in Computer Science, Engineering, or related field (or equivalent)
- Deep understanding of distributed systems and scalable backend architectures
- Hands-on experience building and shipping LLM-powered or GenAI applications
- Understanding of model internals, inference pipelines, evaluation techniques, and prompt engineering
- Ability to thrive in ambiguous, fast-changing environments with product mindset
- Excited to shape AI observability tools
- Clear communicator with rigorous thinking and clean code
- Experience with observability tools/platforms

## Benefits and Growth
- Build tools for software engineers
- Influence product direction and business impact
- Work with skilled teammates
- Competitive global benefits
- Continuous professional development

## Similar roles

- [Staff GenAI Engineer - Application Performance Monitoring](https://hotfix.jobs/jobs/bea56c44-c929-42b7-a6be-897d60cf826f) - Datadog - New York, NY - $234k – $234k/yr
- [Staff AI Engineer - Cortex Code Quality](https://hotfix.jobs/jobs/1274746b-863d-4937-84f2-c41ac854c1b9) - Snowflake - Menlo Park, CA - $236k – $339k/yr
- [Staff Software Engineer, Agents](https://hotfix.jobs/jobs/a97dd7fb-bd69-4449-8010-971ab051bbfa) - Harvey - San Francisco, CA - $231k – $340k/yr
- [Senior Staff Software Engineer, AI Model LifeCycle](https://hotfix.jobs/jobs/75876f8d-3aba-4f82-9049-17169f5eec63) - Crusoe - San Francisco, CA - $238k – $318k/yr
- [Staff Machine Learning Engineer](https://hotfix.jobs/jobs/ebc73812-e806-404d-9f4e-435da967dbf5) - Reddit - Remote - $230k – $322k/yr

**Apply:** https://hotfix.jobs/jobs/29e1a9d6-f50c-4626-8874-48c117a5c52e
**Canonical:** https://hotfix.jobs/jobs/29e1a9d6-f50c-4626-8874-48c117a5c52e