Senior Infra Engineer: Observability

Responsibilities

• Build ingestion pipelines to consume 1M+ RPS streams of logs, metrics, and other telemetry
• Build scalable, fault tolerant alerting engines for notifying users, in real-time, of threshold breaches
• Craft rich backend observability APIs, working with product to build amazing experiences for instantly grokking their application
• Provide APIs to access realtime log/metrics streams to be consumed by the Dashboard and Product Teams
• Build Golang/Rust GRPC services from scratch capable of supporting tens of thousands of users
• Define infrastructure that can be torn down, failed over, and reconstituted from scratch using principle of immutable infrastructure using Terraform and Ansible
• Write Engineering Requirement Documents to take something from idea, to defined tasks, to implementation, to monitoring it’s success
• Interface with our TypeScript and GraphQL edge to expose your microservice APIs for both internal and potentially external consumption

Requirements

• Strong understanding of distributed systems
• Enjoy building fault tolerant, resilient, and scalable services
• Interests in VictoriaMetrics, ClickHouse, and other systems for building observability stacks from the ground up
• Solid intuition about how long your solutions will last
• Implement solutions, create monitors for error boundaries, and document requirements
• Great sense of direction and prioritization when dealing with ambiguity of an early stage startup
• Sense of grit to dive into a problem, implement a solution, scale that solution, and replace it when needed
• Great communication skills