Senior Staff GenAI Engineer - Application Performance Monitoring (APM)
Lead GenAI/ML initiatives for Datadog's APM team, designing and deploying models for agentic workflows, automated investigations, and performance optimization. Requires 10+ years experience with 4+ years leading cross-team projects.
About the job
What You’ll Do:
- Serve as technical owner for GenAI initiatives within APM, leading design, development, and deployment of ML/AI-powered features across multiple teams.
- Guide long-term strategy and technical direction for GenAI workflows across APM and related products.
- Build and benchmark GenAI/ML models using state-of-the-art techniques.
- Contribute to Datadog’s broader senior engineering community through thought leadership and collaboration.
- Collaborate with cross-functional teams to build automated investigation and triaging tools.
- Influence product direction with a strong product mindset, advocating for end users.
- Guide teams through ambiguity, scaling challenges, and evolving requirements.
- Actively mentor engineers and influence engineering culture through design reviews, technical talks, and working groups.
Who You Are:
- BS/MS/PhD in a scientific field or equivalent experience.
- 10+ years of relevant engineering experience, including 4+ years leading cross-team technical initiatives.
- Proven track record of leading large-scale GenAI/ML initiatives in a product-driven environment.
- Highly proficient in model development, deployment, evaluation, and optimization, with hands-on experience in end-to-end ML systems.
- Ability to drive initiatives across cross-functional teams and solve ambiguous challenges.
- Passionate about mentoring and cultivating high-performing engineering culture.
- Excited about leveraging AI tools to enhance coding, problem-solving, and building.
Benefits and Growth:
- Build tools for software engineers and use them to accelerate development.
- Influence product direction and business impact.
- Work with skilled, knowledgeable teammates.
- Competitive global benefits.
- Continuous professional development.
Skills
Generative AI, Machine Learning, Ml Models, Distributed Tracing, Model Deployment, Model Evaluation, Model Optimization, Python, Kubernetes, LLMs
Similar jobs
ML Engineering jobs
Square
San Francisco, CA
Build and operate production machine learning systems for ranking, retrieval, recommendations, personalization, and customer intelligence. The role requires 12+ years of production software and ML experience, strong expertise in intelligent systems, and sound judgment around trustworthy customer-impacting signals.
United States
Leads the technical direction and development of large-scale, GenAI-powered recommendation and feed-ranking systems. Requires 10+ years of industry experience in relevance-driven products, deep expertise in machine learning and recommendations, and strong organizational influence and mentoring skills.
United States
Leads the technical direction of large-scale ML infrastructure for embedding, recommendation, and personalization systems. The role requires 8+ years of ML engineering experience, expertise in deep learning and distributed training, and strong leadership across research, infrastructure, and production deployment.
Scale AI
San Francisco, CAStaff engineer responsible for designing and scaling the infrastructure, execution environments, verifiers, and tooling used to train and evaluate AI agents. Requires 8+ years of software engineering experience, strong Python and distributed-systems expertise, and familiarity with sandboxing, high-throughput systems, and LLM workflows.
United States
Leads technical strategy for Reddit’s Ads ML Platform, improving feature development, training-data generation, experimentation, and the path to production ML serving. The role requires 8+ years in infrastructure or distributed systems, production ML platform experience, and strong cross-team technical leadership.