Staff, Backend Engineer - Catalog
Leads development of DataHub's platform framework, building scalable metadata ingestion systems, APIs, event-driven processing, schema mapping, and AI asset versioning. Requires 8+ years in distributed systems, advanced Python/API expertise, and high-scale data processing experience.
Responsibilities
- Build scalable, fault-tolerant ingestion systems for enterprise-scale metadata
- Develop clean, intuitive APIs for our connector ecosystem
- Implement event-driven architectures for real-time metadata processing
- Create schema mapping between diverse systems and DataHub's unified model
- Design versioning systems for AI assets (training data, model weights, embeddings)
Requirements
- 8+ years building production-grade distributed systems
- Advanced Python and API design expertise
- Experience with high-scale data processing or integration frameworks
- Strong systems knowledge and distributed architecture experience
- Proven track record solving complex technical challenges
- Built and maintained online applications serving live traffic at scale (100+ QPS)
- Set up monitoring and alerting for services
- Designed indexing, storage, and data architectures to make large-scale data accessible to online services
- Designed and scaled distributed systems
- Hands-on experience developing in a tight loop with LLMs and applying best practices for scalable LLM development
Languages
- One of Java/Scala/Kotlin/C#/Go (very strong nice-to-have / borderline must-have)
- Python/TypeScript/Node.js (nice-to-have)
Technical Skills
- AWS
- Kubernetes/Docker
- CI/CD deployment pipelines
- Microservice Architecture
Nice-to-Haves
- Experience with DataHub or similar metadata/ETL frameworks (Airflow, Airbyte, dbt)
- Open-source contributions
- Experience building and maintaining services that make calls to LLMs in order to serve live traffic
- Experience fine-tuning LLM-powered applications exposed to end users
- Early-stage startup experience
Compensation
Salary Range: $225,000 to $300,000
Principal Software Engineer, SIEM
Principal Software Engineer leading a feature delivery team to design, develop, and release software for the Huntress SIEM platform. Requires 15+ years experience, 8+ years Ruby on Rails and Postgres scaling.
Principal Software Engineer, Cash Line
Principal backend engineer building core systems for a new AI-powered revolving credit product. Owns distributed services for underwriting, pricing, transactions, and servicing in a greenfield environment.
Staff Software Engineer
Staff-level engineer building and scaling a new Identity Security Posture Management product. Owns technical vision, architecture for petabyte-scale data systems, and mentors junior engineers using Ruby on Rails.
Staff Software Engineer
Staff Software Engineer building and scaling a new Endpoint Security Posture Management platform from the ground up. Focus on Ruby on Rails, data pipelines, and petabyte-scale systems while mentoring junior engineers.