Responsibilities
- Lead the development and operation of a data lake for cyber operations and intelligence data.
- Design schemas, partitions, and indexes that make complex datasets performant and cost-effective to query.
- Partner with engineers and intelligence analysts to define query patterns and data products for mission use cases.
- Build and evolve ETL pipelines that are observable, recoverable, and resilient to upstream change.
- Drive technical initiatives end-to-end, from architecture decisions through production rollout and iteration.
- Establish best practices for data quality, documentation, and operational ownership across the platform.
- Mentor engineers on data modeling, performance tuning, and production-grade pipeline design.
- Identify bottlenecks in storage/compute/query layers and ship improvements with clear performance wins.
Requirements
- 8+ years of experience in data engineering and/or data architecture.
- Mastery-level expertise building ETL pipelines and operating them in production.
- Deep experience with data lake architecture and systems used to query data lakes.
- Strong schema and index design skills, including partitioning, indexing, and clustering strategies.
- Experience with column-oriented databases in production environments.
- Built data systems from scratch (not only maintained existing platforms).
- Proven leadership experience mentoring engineers and driving technical initiatives.
- U.S. citizen with active TS/SCI security clearance with appropriate polygraph.
Nice To Haves
- Experience with key-value datastores.
- Worked with streaming and message queue systems.
- Experience with graph database technologies.
- Worked with internet/networking datasets (e.g., scan data, DNS, netflow, certificates).
- Experience supporting analysts or operational users with high-stakes data needs.
Tech Stack
Data lakes: Apache Iceberg, Delta Lake, Apache Hive
Query engines: Trino, Presto, AWS Athena, Apache Spark
Column stores: ClickHouse, Amazon Redshift, Google BigQuery
ETL / orchestration: Airflow, AWS Glue, NiFi, ClickPipe
Streaming / queues: Kafka, RabbitMQ, NATS, AWS Kinesis
Graph: Neo4j, AWS Neptune, Memgraph, Apache AGE