# Principal Software Engineer, Applied AI

**Company:** [Invisible Tech](https://hotfix.jobs/companies/invisible-tech)
**Location:** New York, NY
**Role:** Backend Engineering
**Salary:** $250k – $366k/yr
**Experience:** 8+ years
**Skills:** Python, LLMs, AI Agents, RAG, MLOps, Kubernetes, AWS, GCP, Azure, DevOps
**Posted:** 2026-03-05

> Principal Software Engineer builds scalable backend systems for AI/ML operations, deploys production ML models, and collaborates with clients on use case discovery and AI solutions. Requires 8+ years experience in ML engineering, Python, cloud platforms, and client-facing work.

## Job Description

## What You’ll Do

- As part of the Forward Deployed Engineering team, contribute during rapid growth by scaling models, improving platform performance, building backend systems, supporting client-facing deployments, and enabling smoother workflows for machine learning solutions.
- Develop and maintain AI/ML systems: Build robust, scalable backend systems that support machine learning operations and data processing pipelines.
- Cloud operations and management: Oversee and optimize cloud infrastructure to ensure efficient deployment and operation of ML models.
- Problem solving: Independently explore and address complex problem spaces to improve system capabilities and performance without extensive guidance.
- Cross-functional collaboration: Work closely with ML engineers and data scientists to integrate advanced ML technologies, ensuring seamless operations across various platforms.
- Client engagement: Collaborate directly with clients, working embedded with client teams to support use case discovery, product development, and AI deployment.
- Innovation and R&D: Actively participate in research and development of new tools that can enhance AI capabilities and workflows.

## What We Need

- 8+ years of software engineering experience, with a strong focus on ML engineering and deploying machine learning models in production.
- Extensive experience in full-stack development, particularly in backend environments that support AI/ML workloads.
- Prior experience working directly with clients in use case discovery, product development, and leading client engagements.
- **Technical Expertise:**
  - Strong proficiency in **Python**, with deep expertise in **LLMs**, **AI Agents**, and ML model development.
  - Experience designing and deploying scalable ML systems, such as **retrieval-augmented generation (RAG)** pipelines and production-grade AI applications.
  - Extensive experience with cloud platforms (**AWS**, **GCP**, **Azure**) and operational best practices for ML workloads.
  - Familiarity with **Kubernetes** and other container management tools.
  - Ability to write well-structured, organized code and automated unit/E2E tests.
  - Comfortable with polyglot persistence models (**SQL** vs. **NoSQL**).
- **ML Operations:** Experience with **MLOps** frameworks and best practices; familiarity with **DevOps** principles as applied to machine learning models, including model versioning, monitoring, and lifecycle management.
- Problem solving: Ability to operate independently in unstructured environments, demonstrating a proactive and investigative approach to tackling challenges.
- Communication: Excellent communication skills, with the ability to collaborate effectively in dynamic, cross-functional teams, including data scientists, researchers, and software engineers.

## Similar roles

- [Principal Software Engineer, Money Infrastructure](https://hotfix.jobs/jobs/2f82d675-f4ad-4c53-b685-c1a5f6532852) - Replit - Foster City, CA - $250k – $340k/yr
- [Principal Software Development Engineer Person & Trust Platform](https://hotfix.jobs/jobs/c1c4626b-5a98-4458-99d3-a7cf01058f76) - Idme - Mountain View, CA - $243k – $307k/yr
- [Principal Software Engineer I - Metadata](https://hotfix.jobs/jobs/8851fa3b-008f-4f5e-93f9-ca01a0ea30b6) - Snowflake - Menlo Park, CA - $264k – $380k/yr
- [Principal Software Engineer - Snowflake Intelligence](https://hotfix.jobs/jobs/160814ee-9a10-4deb-99a4-22b0bc45be91) - Snowflake - Menlo Park, CA - $264k – $380k/yr
- [Principal Software Engineer I - Snowhouse Foundation](https://hotfix.jobs/jobs/eab67d12-c617-42e1-9ef4-0e6de3eb7c9c) - Snowflake - Menlo Park, CA - $264k – $380k/yr

**Apply:** https://hotfix.jobs/jobs/b2a2e9b1-4cc0-417b-b015-df109bc4c6ca
**Canonical:** https://hotfix.jobs/jobs/b2a2e9b1-4cc0-417b-b015-df109bc4c6ca