Software Engineer - Hosted Model Infrastructure

Core Responsibilities

• Building high-performance model serving infrastructure that integrates with security models, hardware constraints, and different inference engines
• Designing intelligent request handling including authentication, rate limiting, concurrency control, and audit logging for multi-tenant model access
• Building and maintaining packaging and deployment pipelines enabling fast, secure, and reliable model rollouts across on-premises and air-gapped environments
• Developing observability for production AI systems to enable easy service monitoring and fast incident triage and resolution
• Debugging complex issues and performance problems throughout the stack, including open source inference engines, container runtimes, and GPU drivers, in environments you cannot always access directly
• Designing and running testing and benchmarking infrastructure that validates model deployments across varying GPU hardware before they reach production
• Working with product teams and customers to understand requirements, debug production issues, and deliver the models and capabilities they need
• Integrating hosted model infrastructure with Palantir's deployment, configuration, and identity systems

What We Value

• Ownership mindset and bias toward quality. Our software runs in environments where direct access for debugging is limited or unavailable.
• High empathy for customer needs and drive to deliver reliable, easy-to-use models
• Ability to work effectively across multiple languages and layers of the stack, from backend services and ML tooling to container orchestration and deployment configuration
• Strong debugging skills and motivation to trace problems from application code through containers, orchestration, and hardware
• Curiosity about emerging AI capabilities and the ability to quickly evaluate and integrate new models and technologies as the landscape evolves
• Active US Security clearance, or eligibility and willingness to obtain a US Security clearance is beneficial, but not necessary

What We Require

• 4+ years of professional software engineering experience building and operating production systems
• Engineering background in Computer Science, Mathematics, Software Engineering, Physics, or similar field
• Strong coding skills with demonstrated proficiency in programming languages, such as Java, C++, Python, Rust, or similar languages. Familiarity with the Python ML ecosystem is valuable.
• Experience with containers, Kubernetes, and deploying backend services in production environments
• Strong written and verbal communication skills and ability to iterate quickly with teammates, incorporating feedback and holding a high bar for quality