Data Engineering Manager

Key Responsibilities

Market Data Pipeline Engineering

• Own the market data pipeline from ingestion through normalization and near-real-time delivery. The data has to be correct first, and the system has to recover cleanly when something breaks.
• Integrating direct exchange feed capture alongside third-party vendor data
• Building and improving replay, recovery, and gap-detection capabilities
• Keeping market data correctly sequenced, validated, and available fast enough for downstream users
• Understanding when latency matters, when durability matters more, and how to make the right tradeoff

Time-Series Architecture: KDB+/Q

• Design, maintain, and improve the KDB+/Q platforms that hold our real-time and historical market data.
• Schema design, partitioning, and query-performance tuning
• Supporting real-time and historical analytics use cases
• Managing retention and data lifecycle policies
• Keeping the platform maintainable as data volumes and usage grow
• Debugging production HDB and tickerplant issues directly

Data Distribution & Platform Integration

• Deliver data reliably to downstream consumers through streaming, messaging, and platform integrations.
• Defining data contracts and schemas that other teams can depend on
• Supporting replayable and durable data flows where needed
• Working with downstream teams to understand how they actually consume the data
• Balancing real-time delivery needs with reliability and operational simplicity

Tooling, Libraries & Supporting Systems

• Build the internal tooling and shared libraries that make the data platform easier to operate and easier to use.
• Building validation, monitoring, replay, and analytics tools
• Owning supporting systems for reference data, configuration, and metadata
• Improving developer workflows around market data testing and troubleshooting
• Reducing repeated manual work through better tools and automation

Technical Leadership & Production Ownership

• Lead the team by staying close to the work.
• Writing production code
• Reviewing pull requests and technical designs
• Working directly with trading and research teams to understand their needs
• Debugging production issues during market hours when needed
• Setting expectations for quality, reliability, and maintainability
• Improving monitoring, alerting, and data-quality checks so problems are caught before the desk finds them

Technology Stack

• KDB+ / Q
• Python
• C / C++
• Linux
• Docker
• Git / CI-CD
• Binary market data protocols
• Streaming / message bus platforms
• Kernel-bypass / high-performance networking
• Industry-standard messaging protocols (FIX, SBE)

Required Qualifications

• At least 7 years of experience in data engineering, market data infrastructure, or a closely related area
• Current hands-on production coding experience
• At least 3 years leading engineers while staying technically involved
• Strong KDB+/Q experience, including complex Q, tick architecture, query tuning, and production HDB troubleshooting
• Strong production Python experience, including tested, packaged, maintainable systems-level code
• Experience building low-latency decoders for real exchange protocols
• Strong understanding of multicast, packet capture, sequencing, and gap detection
• Comfortable working in Linux and using tools such as perf, strace, tcpdump, and numactl
• Able to own production issues directly, not just route them to someone else

Preferred Qualifications

• Background in high-frequency trading, market making, proprietary trading, or another latency-sensitive environment
• C or C++ experience for performance-critical decoder or capture components
• Experience with kernel-bypass or high-performance networking technologies
• Experience with streaming platforms used in real-time data pipelines
• Working knowledge of binary market data encoding standards