Staff Software Engineer - ML Observability
Lead development of LLM observability features at Datadog, building tools for monitoring, tracing, and evaluating AI systems in production. Requires expertise in distributed systems, GenAI applications, and model internals.
About the job
What You’ll Do
- Drive design and implementation of LLM observability features
- Ideate, prototype, and scale new product features for generative AI systems
- Work cross-functionally with engineering, product, UX, and applied science teams
- Develop tools for tracing, evaluating, and debugging LLMs
- Influence architecture decisions and mentor engineers
- Stay close to customer pain points to guide priorities
- Stay current with ML and observability trends
Who You Are
- BS/MS/PhD in Computer Science, Engineering, or related field (or equivalent)
- Deep understanding of distributed systems and scalable backend architectures
- Hands-on experience building and shipping LLM-powered or GenAI applications
- Understanding of model internals, inference pipelines, evaluation techniques, and prompt engineering
- Ability to thrive in ambiguous, fast-changing environments with product mindset
- Excited to shape AI observability tools
- Clear communicator with rigorous thinking and clean code
- Experience with observability tools/platforms
Benefits and Growth
- Build tools for software engineers
- Influence product direction and business impact
- Work with skilled teammates
- Competitive global benefits
- Continuous professional development
Skills
LLMs, Generative AI, Distributed Systems, Backend Architectures, Model Inference, Prompt Engineering, Observability Tools, Llm Tracing, Model Evaluation, Drift Detection
Similar jobs
ML Engineering jobs
Shield AI
San Mateo, CA
Leads the design, implementation, integration, and field validation of tactical autonomy and multi-agent coordination capabilities for unmanned platforms. Requires 7+ years of relevant experience, production C++, technical leadership, and eligibility for a U.S. Secret clearance.
Snowflake
Bellevue, WA
Leads the roadmap and technical vision for Snowflake Feature Store, building reliable, high-performance machine learning platform capabilities and supporting technical execution across partner teams. Requires 10+ years of experience with data-serving infrastructure or ML platforms, plus Java and Python expertise.
Harvey
San Francisco, CA
Leads the design and operation of reliable, scalable model infrastructure powering AI inference across multiple providers. Requires 7+ years of distributed-systems engineering experience, strong programming skills, and expertise in production reliability and cloud infrastructure.
Shield AI
Washington, DCLeads development and integration of advanced maritime autonomy for USVs, UUVs, and cooperating UAVs, including motion planning, localization, safety, and multi-agent coordination. Requires staff-level technical leadership, substantial robotics experience, C++ and Python proficiency, and eligibility for a SECRET clearance.
Airbnb
United States
Senior Staff ML Engineer fine-tunes and optimizes state-of-the-art LLMs for Airbnb's customer support AI products, including AI assistants and autonomous agents. Partners cross-functionally to productionize models at scale. Requires PhD and 10+ years experience with PyTorch.