# Senior AI Engineer - APM Experiences

**Company:** [Datadog](https://hotfix.jobs/companies/datadog)
**Location:** New York, NY
**Role:** ML Engineering
**Salary:** $187k – $240k/yr
**Experience:** 4+ years
**Skills:** LLMs, AI Agents, Go, Java, Python, Distributed Tracing, OpenTelemetry, RAG, Machine Learning, Observability
**Posted:** 2026-01-21

> Leads development of LLM- and agent-based features for Datadog's APM to detect, resolve, and prevent performance issues using traces, metrics, and logs. Requires 4+ years in backend/ML systems, production AI experience, and distributed systems expertise.

## Job Description

## The opportunity

Datadog’s APM Experiences team owns the core product experience for Application Performance Monitoring — including distributed tracing, service representation, and more. We’re building a new wave of AI-powered capabilities that help customers *detect, resolve, and prevent* performance issues faster. In this role, you will lead end‑to‑end development of LLM- and Agent‑based features that can:

* Debug and investigate application performance issues down to the root cause, as both a developer assistant and a fully autonomous agent
* Proactively recommend performance and reliability-based optimizations to prevent the next incident
* Automatically create intelligent monitors and SLOs for the most important business flows and critical paths

This is a highly product‑minded engineering role: you’ll work from problem discovery and UX all the way to reliable, scalable production systems.

## What you’ll do

* **Shape AI experiences for APM.** Design and ship LLM/agentic workflows that analyze traces, metrics, logs, and other telemetry to generate diagnoses, explanations, and guided fixes.
* **Own the full loop.** Prototype quickly, define success metrics and evals, run experiments, iterate, and ultimately productionize for scale and reliability.
* **Build robust agent systems.** Develop tools, retrieval and planning strategies, and guardrails; manage prompts/evals; design fallbacks and human‑in‑the‑loop paths.
* **Integrate with Datadog’s platform.** Leverage surfaces like Trace Explorer, Service Catalog, monitors, and workflows to deliver end‑to‑end value in the APM UI.
* **Partner deeply.** Collaborate with PM, Design, and partner teams to build cohesive experiences.
* **Raise the bar on engineering.** Write performant, maintainable backend code, own services in production, and improve reliability for high‑throughput, low‑latency data systems.

## Who you are

**Product‑minded engineer who ships AI to production**

* 4+ years building backend or real-time ML systems; you value simplicity, correctness, and performance
* Proven experience delivering LLM/agent features to production (prompting, tooling, evals, safety/guardrails)
* Comfortable owning user journeys, iterating from prototype → alpha → GA, and measuring impact with clear product metrics

**Strong ML / applied science fundamentals**

* Solid grasp of the ML lifecycle (task definition, dataset collection, modeling, evaluation, deployment, iteration) and statistics (experiment design, confidence intervals)
* Experience choosing/modeling the right technique for the job (e.g., anomaly detection, ranking/recommendation, NLP), and knowing when a heuristic beats a model
* Fluency with offline/online evals for AI systems; can build reliable golden sets and automatic regressions

**Distributed systems & observability savvy**

* Experience with microservices performance: tracing, latency breakdowns, concurrency, and resiliency patterns
* Proficient in Go, Java, or Python; strong API/service design; production ops (monitoring, alerting, on‑call rotation)

**Nice to have**

* Hands‑on with distributed tracing stacks (OpenTelemetry/Datadog APM), profilers, and logs/metrics pipelines
* Exposure to planning/agent frameworks, tool‑use orchestration, RAG, and retrieval/indexing for observability data
* Familiarity with SLO/SLA practices and incident response

## Similar roles

- [Senior Software Engineer - AI Code Gen - Agent Engineer](https://hotfix.jobs/jobs/29a07c60-76fb-45d5-bb91-449a9b2ff20d) - Datadog - New York, NY - $187k – $240k/yr
- [Senior Software Engineer - AI Platform Team](https://hotfix.jobs/jobs/24cb8d81-486e-4cf5-b3fe-a940677eeced) - Coinbase - Remote - $186k – $219k/yr
- [Sr Software Engineer, Cribl AI](https://hotfix.jobs/jobs/db813bb7-e0aa-4356-accd-71a930299507) - Cribl - Remote - $185k – $215k/yr
- [Sr Machine Learning Engineer, AI Research](https://hotfix.jobs/jobs/dd21fdef-4c0a-480d-916f-128ec97de545) - Cribl - Remote - $185k – $215k/yr
- [Senior Machine Learning Engineer - Scene Understanding](https://hotfix.jobs/jobs/b97a5d06-e2a8-4276-bb59-e56092a0a769) - Zoox - Foster City, CA - $189k – $290k/yr

**Apply:** https://hotfix.jobs/jobs/b1e83c7d-5af6-4fd9-ad94-af60f435f81a
**Canonical:** https://hotfix.jobs/jobs/b1e83c7d-5af6-4fd9-ad94-af60f435f81a