Lead Software Platform Engineer, MLOps

Responsibilities

• Own the technical architecture of the AI/ML platform and the service and API surface used to run models and agents against scientific data.
• Own the end-to-end model and prompt lifecycle across Databricks MLflow and AWS Bedrock, including registration, versioning, asset bundles, staged promotion, rollback, and multi-model serving.
• Design inference infrastructure for real-time and batch workloads, including routing, batching, caching, concurrency control, GPU and accelerator capacity planning, large binary inputs, and graceful degradation.
• Integrate AI models and LLMs into production systems using retrieval-augmented generation (RAG), tool and function calling, MCP-based tooling, and agent runtimes.
• Design security controls for guardrails, prompt-injection and tool-abuse defenses, PII and PHI handling, and tenant data boundaries.
• Build evaluation and quality infrastructure, including offline and online evaluation harnesses, golden datasets, CI regression gates, A/B and shadow deployments, and drift and hallucination detection.
• Establish monitoring, alerting, logging, and distributed tracing, and define SLI, SLO, and SLA practices for probabilistic systems.
• Design reproducibility and lineage for validated environments through versioned data, code, prompts, and model artifacts and auditable trails.
• Contribute to infrastructure-as-code and deployment automation using CloudFormation and AWS CDK, including multi-tenant infrastructure, online upgrades, and on-demand compute allocation.
• Own production readiness, performance, reliability, cost efficiency, incident response, and runbooks for the AI platform.
• Lead design reviews, write reference architectures and technical documentation, mentor engineers, and evaluate emerging AI infrastructure and build-versus-buy decisions.

Requirements

• 10+ years of professional experience in software and infrastructure engineering, designing, building, and scaling distributed cloud-native systems in production.
• Experience as a technical leader or architect accountable for system design, scalability, performance, and cost optimization.
• Experience designing security into multi-tenant platforms, including tenant authorization boundaries, PII and PHI handling, prompt injection, and tool-abuse risks.
• Extensive experience building and maintaining production AI/ML infrastructure as a multi-tenant product for external users.
• Hands-on experience taking LLM systems to production, including RAG, retrieval and embedding design, prompt and model versioning, and tool or function calling.
• Expert coding skills in TypeScript and Python for robust APIs and backend services.
• Production experience with model registries and serving stacks, ideally Databricks MLflow.
• Experience using AI evaluation as a release gate, including evaluation harnesses, regression gates, and drift or quality monitoring.
• Proficiency in API-first design, REST, and OpenAPI.
• Working knowledge of AWS and Docker, plus infrastructure-as-code such as CloudFormation or AWS CDK.
• Experience with CI/CD pipelines, deployment automation, monitoring, alerting, distributed tracing, and SLI/SLO/SLA practices.
• Strong communication, cross-functional influence, technical direction, and mentoring skills.