Software Engineer AI Gateway
Build and maintain distributed server-side software for Kong’s AI Gateway, including LLM routing, proxying, load balancing, and semantic caching. The role requires 3+ years of server-side programming experience, Go or Rust expertise, strong Linux and networking skills, and familiarity with AI/LLM applications.
About the job
Responsibilities
- Build and maintain API management software for AI and large language models (LLMs) and its underlying stack.
- Implement features including proxying capabilities, load balancing, and database support for the Kong AI Gateway.
- Troubleshoot bugs, tune performance, and perform low-level optimizations.
- Integrate Kong with third-party deployment platforms and tooling.
- Work on multi-LLM routing and balancing, semantic caching and routing, MCP, and related AI Gateway capabilities.
- Complete additional tasks assigned by the manager.
Requirements
- 3+ years of programming experience with server-side applications and components.
- Hands-on experience with Go or Rust.
- Experience building and debugging highly scalable, distributed, reliable, and resilient applications.
- Strong low-level technical expertise in Linux systems troubleshooting and profiling.
- Experience with distributed systems, the Linux networking stack, and computer systems programming.
- Knowledge of APIs across L7 and L4 transports and protocols.
- Familiarity with AI/LLM concepts and application development.
- Comfortable working with a remotely distributed team.
- Strong sense of ownership and excellent verbal and written communication skills.
- Ability to write product documentation.
Nice-to-haves
- Hands-on experience with NGINX, OpenResty, Kong, or LuaJIT.
- Hands-on experience with Envoy or Istio.
Skills
Go, Rust, Linux, Distributed Systems, Linux Networking, Computer Systems Programming, Api Management, Load Balancing, Nginx, Openresty, Kong, Luajit, Envoy, Istio, Llm Applications
Similar jobs
Backend Engineering jobs
Rain
Remote
Design and lead secure, scalable wallet infrastructure spanning custody, key management, signing, authorization, and recovery. The role requires deep expertise in wallet or cryptographic security infrastructure, distributed systems architecture, and modern blockchain account models.
Supabase
Remote
Build and operate a globally distributed CI/CD execution platform, including branch orchestration, ephemeral build environments, scheduling, caching, and pipeline observability. Requires Go and TypeScript expertise plus at least 3 years building cloud infrastructure or distributed systems.
Hudl
Chiavari, Italy
Builds and operates scalable, backend-heavy data APIs and services for Hudl’s global football and rugby products. The role requires 3+ years of backend or data engineering experience, API and cloud expertise, and comfort with customer-facing product development.
Phantom
Remote
Senior backend engineer designing scalable services and APIs that connect Phantom Cash to blockchain networks. The role requires Node.js/NestJS, REST API, event-driven, and distributed-systems expertise, with blockchain or fintech experience preferred.
Fingerprint
Remote
Build and operate highly performant Core API services for real-time fraud detection, owning reliability, observability, scaling, and production operations. The role requires 5+ years of backend experience with Go or Node.js and strong SRE practices.