Senior Machine Learning Engineer, Applied AI Quality
Senior ML Engineer leading architecture and strategy for LLM- and agent-powered quality evaluation systems at Block. Builds scalable AI frameworks to measure product behavior, detect issues, and generate insights across millions of interactions to improve reliability and decision-making.
About the job
You Will
- Lead the technical strategy and architecture for AI-driven quality and evaluation systems used across products and teams.
- Drive the development of scalable systems that use LLMs, agents, and behavioral signals to evaluate quality, detect regressions, and generate product insights.
- Define long-term approaches for evaluation, measurement, and quality intelligence across complex product surfaces.
- Translate ambiguous organizational needs into clear technical direction, roadmap priorities, and platform capabilities.
- Influence engineering standards and best practices for building reliable, measurable, and trustworthy AI systems.
- Lead complex cross-functional initiatives spanning product, infrastructure, data, and applied AI teams.
- Mentor and level up engineers across the organization through technical leadership, design reviews, and systems thinking.
- Identify leverage opportunities where AI systems can fundamentally improve how teams understand, debug, and improve product behavior.
You Have
- 5+ years of experience in software engineering, machine learning engineering or applied AI.
- Deep experience designing and shipping large-scale AI/ML systems in production environments.
- Strong expertise with LLMs, agents, evaluation systems, retrieval architectures, and modern AI infrastructure.
- Proven ability to lead ambiguous, high-impact technical initiatives from concept through adoption across multiple teams.
- Strong systems thinking and architectural judgment, with the ability to balance experimentation, scalability, and operational rigor.
- Experience defining technical strategy and influencing roadmaps beyond your immediate team.
- Excellent communication and cross-functional leadership skills, with the ability to align engineering, product, and organizational priorities.
- A track record of creating leverage through platforms, frameworks, and systems that enable other engineers and teams to move faster and make better decisions.
Skills
LLMs, AI Agents, Machine Learning, Evaluation Systems, Retrieval Architectures, Python, Large-Scale Ai Systems, AI Infrastructure, Systems Architecture
Similar jobs
ML Engineering jobs
Square
United States
Leads the design, deployment, and optimization of agentic and generative AI systems that automate risk and compliance investigations at scale. Requires 8+ years of machine learning modeling experience, production ML expertise, and advanced technical education.
Nuro
Mountain View, CABuild and operate large-scale infrastructure for autonomous-driving model training, including distributed GPU systems, data pipelines, ML workflows, and reliability tooling. The role requires 3+ years of experience, strong Python and systems-language skills, Kubernetes expertise, and distributed-systems fundamentals.
Shield AI
San Mateo, CA
Design and deploy tactical autonomy algorithms and high-performance software for unmanned systems operating in complex, contested environments. The role requires 5+ years of related experience, strong C++ and Python skills, robotics expertise, and the ability to obtain a SECRET clearance.
Airbnb
United States
Senior machine learning engineer who will build and operate large-scale AI systems for Airbnb’s payments ecosystem, including LLM agents, fraud defenses, and personalization. The role requires 5+ years of applied AI/ML experience, strong Python or Java skills, and production MLOps expertise.
United States
Build and operate large-scale machine learning infrastructure and models for Reddit’s recommendation and personalization systems. The role requires 5+ years of ML engineering experience, expertise in deep learning and distributed systems, and proficiency with Python and modern ML frameworks.