Senior Software Engineer - Observe Data Management
Builds and scales petabyte-scale data ingestion pipelines for observability platform using Go/C++ on AWS/Azure. Requires 5+ years in distributed systems, strong systems programming, and cloud experience.
About the job
Responsibilities
- Design, build, and scale high-throughput data ingestion and processing pipelines handling petabyte-scale telemetry — logs, metrics, traces, and events
- Develop performance-critical, distributed systems components in Go and/or C++ that operate reliably across AWS and Azure
- Contribute to OpenTelemetry and drive Observe's open-source strategy, including external community engagement and upstream contributions
- Architect solutions that maintain enterprise-grade availability and low latency under extreme data volumes
- Collaborate with SRE, product, and platform teams to define data reliability standards and improve detection-to-resolution times for customers
- Debug and resolve complex distributed systems issues at the deepest layers of the stack
- Help shape the technical roadmap for the Data Management team and mentor engineers across the organization
Requirements
- 5+ years of software engineering experience with deep expertise in distributed systems
- Proficiency in Go and/or C++, with an ability to write high-performance, production-grade systems code
- Demonstrated experience designing and operating large-scale data ingestion or stream processing pipelines
- A strong sense of user empathy and product intuition — you think beyond APIs and care about end-to-end data onboarding and management experience
- Hands-on experience building and running services across major cloud providers (AWS and/or Azure)
- Strong fundamentals in systems programming: concurrency, memory management, networking, and I/O
- A track record of solving hard infrastructure or platform engineering problems at scale
- B.S. in Computer Science, Engineering, or equivalent practical experience
Nice-to-Haves
- Experience with OpenTelemetry SDKs, instrumentation, or ecosystem tooling
- Prior open-source contributions or project maintainership
- Familiarity with Apache Iceberg or other open table formats and data lakehouse architectures
- Background in observability, monitoring, or SRE
- Experience with multi-cloud data infrastructure or telemetry platforms at petabyte scale
Skills
Go, C++, Distributed Systems, AWS, Azure, OpenTelemetry, Apache Iceberg, Stream Processing, Systems Programming, Data Pipelines
Similar jobs
Data Engineering jobs
Decagon
San Francisco, CASenior Data Infrastructure Engineer responsible for building and operating reliable, low-latency streaming and batch data systems that support AI products. Requires 5+ years of production data infrastructure experience and expertise with technologies such as Kafka, Flink, ClickHouse, and Terraform.
Snowflake
Senior Software Engineer on Observe by Snowflake’s Data Management team, owning the APIs, schemas, and abstractions for scalable tables, views, and materialized views across streaming telemetry. Requires 5+ years of experience with databases, SQL, streaming or data pipelines, API design, and production distributed systems.
Discord
United States
Build and operate petabyte-scale data infrastructure powering Discord’s insights and products. The role requires 5+ years of software engineering experience, strong programming skills, and experience with large-scale pipelines, streaming, orchestration, or data warehousing.
Harvey
San Francisco, CABuild and lead the central data platform, covering ingestion, warehousing, orchestration, streaming, self-service frameworks, and trust layers. The role requires 5+ years of production data infrastructure experience, strong Python and SQL skills, and expertise with Snowflake and modern data tooling.
Datadog
Boston, MABuild and operate large-scale revenue data pipelines powering billing and cost attribution, while improving reliability, latency, and correctness. The role requires strong Spark and Airflow experience, cross-functional problem-solving, and operational ownership of mission-critical production systems.