Skip to content

Principal Software Engineer, Applied AI

250k – 366kNew York, NYHybrid8+ YOE
Summary

Principal Software Engineer builds scalable backend systems for AI/ML operations, deploys production ML models, and collaborates with clients on use case discovery and AI solutions. Requires 8+ years experience in ML engineering, Python, cloud platforms, and client-facing work.

About the role

What You’ll Do

  • As part of the Forward Deployed Engineering team, contribute during rapid growth by scaling models, improving platform performance, building backend systems, supporting client-facing deployments, and enabling smoother workflows for machine learning solutions.
  • Develop and maintain AI/ML systems: Build robust, scalable backend systems that support machine learning operations and data processing pipelines.
  • Cloud operations and management: Oversee and optimize cloud infrastructure to ensure efficient deployment and operation of ML models.
  • Problem solving: Independently explore and address complex problem spaces to improve system capabilities and performance without extensive guidance.
  • Cross-functional collaboration: Work closely with ML engineers and data scientists to integrate advanced ML technologies, ensuring seamless operations across various platforms.
  • Client engagement: Collaborate directly with clients, working embedded with client teams to support use case discovery, product development, and AI deployment.
  • Innovation and R&D: Actively participate in research and development of new tools that can enhance AI capabilities and workflows.

What We Need

  • 8+ years of software engineering experience, with a strong focus on ML engineering and deploying machine learning models in production.
  • Extensive experience in full-stack development, particularly in backend environments that support AI/ML workloads.
  • Prior experience working directly with clients in use case discovery, product development, and leading client engagements.
  • Technical Expertise:
    • Strong proficiency in Python, with deep expertise in LLMs, AI Agents, and ML model development.
    • Experience designing and deploying scalable ML systems, such as retrieval-augmented generation (RAG) pipelines and production-grade AI applications.
    • Extensive experience with cloud platforms (AWS, GCP, Azure) and operational best practices for ML workloads.
    • Familiarity with Kubernetes and other container management tools.
    • Ability to write well-structured, organized code and automated unit/E2E tests.
    • Comfortable with polyglot persistence models (SQL vs. NoSQL).
  • ML Operations: Experience with MLOps frameworks and best practices; familiarity with DevOps principles as applied to machine learning models, including model versioning, monitoring, and lifecycle management.
  • Problem solving: Ability to operate independently in unstructured environments, demonstrating a proactive and investigative approach to tackling challenges.
  • Communication: Excellent communication skills, with the ability to collaborate effectively in dynamic, cross-functional teams, including data scientists, researchers, and software engineers.
Skills
PythonLLMsAI AgentsRAGMLOpsKubernetesAWSGCPAzureDevOps
Similar roles at this salary range
All Backend Engineering jobs →
Chainguard

Principal Software Engineer

Principal-level engineer setting technical direction for Athena, a distributed vulnerability data clearing house. Owns architecture, data validation, and scaling from prototype to production-grade systems.

220k – 258kUnited StatesBackend EngineeringRemote7+ YOEGoGCP
Idme

Staff Software Engineer - Wallet/Authentication Platform

Staff Software Engineer leading technical direction for ID.me's authentication platform, building sign-in, MFA, and account management infrastructure serving 140M+ users. Requires 8+ years experience, Java/Spring Boot expertise, and AI-assisted development tooling experience.

222k – 258kMountain View, CABackend EngineeringOn-site8+ YOEJavaFIDO2
Traba

Staff Software Engineer

Lead development of core backend services and platform architecture for an AI-powered industrial supply chain startup. Own architectural decisions, CI/CD, and performance optimization while partnering with the CTO on product and infrastructure strategy.

240k – 300kNew York, NY +1Backend EngineeringOn-site7+ YOEKafkaPython
Perplexity

Member of Technical Staff

Build and operate the connector runtime and semantic layer that enables Perplexity agents to securely integrate with hundreds of external systems and enterprise data sources.

220k – 405kSan Francisco, CA +3Backend EngineeringHybrid4+ YOEGoAWS
Reddit

Staff Software Engineer - Content Platform

Lead development of Reddit's Content Platform, owning Tier-0 services and core data models that power feeds, posting, and commenting. Design scalable distributed systems in Go/Python while mentoring engineers and driving platform evolution.

217k – 304kUnited StatesBackend EngineeringRemote7+ YOEGoC++