IT Engineer, Internal AI Infrastructure
Build the shared hosting, model-routing, observability, security, and cost infrastructure that enables employees to deploy governed internal AI applications. The role requires at least five years of software engineering experience, strong platform or backend expertise, and hands-on LLM operations experience.
About the job
Responsibilities
- Design and build the hosting golden path for internal AI applications, bots, and agents, including runtime patterns for long-running workflows, scheduled jobs, and agent execution.
- Build model routing that directs tasks across frontier and lower-cost models based on cost, capability, and policy, with a complete audit trail.
- Establish observability, usage analytics, audit logging, and per-team cost attribution for company-wide AI workloads.
- Integrate enterprise identity, access, and security patterns, including Okta-connected defaults, so internal AI applications are secure from launch.
- Create sanctioned workflows for applications handling customer, financial, or people data.
- Partner with product AI infrastructure teams to share gateways and telemetry rather than building duplicate systems.
- Help shape platform strategy, roadmap, and engineering practices as a founding member of the infrastructure pillar.
Requirements
- 5+ years of software engineering experience, including significant infrastructure, platform, or backend engineering experience.
- Experience building internal platforms or paved paths adopted by engineering teams.
- Strong system design skills and experience architecting resilient, observable production systems.
- Hands-on experience with LLM operations, such as model gateways or routing, agent runtimes, evaluations, or cost and usage instrumentation.
- Experience with enterprise identity and access management, such as Okta, SSO, and secure service-to-service authentication.
- Excellent communication and cross-functional collaboration skills across security, IT, and business teams.
Nice-to-haves
- Experience building internal AI platforms, workspaces, or developer platforms at comparable scale.
- Familiarity with MCP, agent orchestration frameworks, or skill and plugin registry patterns.
- Experience with audit and compliance programs such as SOX or ITGC, or building systems with audit requirements.
- Experience as the first infrastructure hire on a small team and setting standards across a broad scope.
Compensation and Benefits
- Annual base salary: $153,000–$296,000 USD.
- Equity and benefits including health, dental, and vision coverage; retirement contributions; parental leave; family planning support; mental health and wellness benefits; paid time off; holidays; and other company benefits.
Skills
Infrastructure, Platform Engineering, Backend Engineering, System Design, Llm Operations, Model Routing, Agent Runtimes, Observability, Audit Logging, Cost Attribution, Okta, Single Sign-On, Service-To-Service Authentication, Mcp, Sox
Similar jobs
IT Support jobs
Juicebox
San Francisco, CA
Builds and manages Juicebox’s internal IT foundation, including identity and access management, provisioning, device management, compliance support, and employee troubleshooting. The role requires at least two years of IT or systems administration experience and hands-on Okta or similar platform expertise.
Vanta
Remote
Owns Workday design, configuration, administration, and continuous improvement across People, Payroll, Finance, IT, and Legal. Requires 5+ years of HRIS or People Systems experience, deep expertise in Payroll, Benefits, or Absence, and strong Workday security and business-process knowledge.
Mercor
San Francisco, CAOwns IT operations, incident response, identity, endpoint, and infrastructure reliability for technical teams. The role requires 5–7 years of IT operations or technical support experience, strong root-cause analysis, and expertise with Apple, MDM, Google Workspace, and Okta environments.
Collective Intelligence Project
San Francisco, CA
Provides advanced Mac fleet, identity, network, SaaS, and service-desk engineering while automating IT operations and delivering onsite support. Requires 3–5 years of IT experience, macOS command-line expertise, MDM and identity administration, scripting, and networking skills.
Ramp
San Francisco, CA
Provides hands-on IT support at Ramp’s San Francisco office while administering core SaaS applications, identity workflows, hardware, licenses, and integrations. Requires at least four years of technical experience spanning enterprise deskside support and SaaS administration.