Staff Data Engineer - TS/SCI Cleared
Leads development of high-performance data lakes and ETL pipelines for petabyte-scale cyber operations data. Partners with engineers and analysts to build reliable data systems, drives technical initiatives, and mentors team members. Requires 8+ years experience and TS/SCI clearance.
About the job
Responsibilities
- Lead the development and operation of a data lake for cyber operations and intelligence data.
- Design schemas, partitions, and indexes that make complex datasets performant and cost-effective to query.
- Partner with engineers and intelligence analysts to define query patterns and data products for mission use cases.
- Build and evolve ETL pipelines that are observable, recoverable, and resilient to upstream change.
- Drive technical initiatives end-to-end, from architecture decisions through production rollout and iteration.
- Establish best practices for data quality, documentation, and operational ownership across the platform.
- Mentor engineers on data modeling, performance tuning, and production-grade pipeline design.
- Identify bottlenecks in storage/compute/query layers and ship improvements with clear performance wins.
Requirements
- 8+ years of experience in data engineering and/or data architecture.
- Mastery-level expertise building ETL pipelines and operating them in production.
- Deep experience with data lake architecture and systems used to query data lakes.
- Strong schema and index design skills, including partitioning, indexing, and clustering strategies.
- Experience with column-oriented databases in production environments.
- Built data systems from scratch (not only maintained existing platforms).
- Proven leadership experience mentoring engineers and driving technical initiatives.
- U.S. citizen with active TS/SCI security clearance with appropriate polygraph.
Nice To Haves
- Experience with key-value datastores.
- Worked with streaming and message queue systems.
- Experience with graph database technologies.
- Worked with internet/networking datasets (e.g., scan data, DNS, netflow, certificates).
- Experience supporting analysts or operational users with high-stakes data needs.
Tech Stack
Data lakes: Apache Iceberg, Delta Lake, Apache Hive
Query engines: Trino, Presto, AWS Athena, Apache Spark
Column stores: ClickHouse, Amazon Redshift, Google BigQuery
ETL / orchestration: Airflow, AWS Glue, NiFi, ClickPipe
Streaming / queues: Kafka, RabbitMQ, NATS, AWS Kinesis
Graph: Neo4j, AWS Neptune, Memgraph, Apache AGE
Skills
Apache Iceberg, Delta Lake, Apache Hive, Trino, Presto, Aws Athena, Spark, ClickHouse, Amazon Redshift, Google Bigquery, Airflow, Aws Glue, Apache Nifi, Kafka, Etl Pipelines
Similar jobs
Data Engineering jobs
SmithRx
United States
Leads enterprise data engineering strategy, architecture, delivery, governance, and technical leadership across the organization. Requires extensive data engineering experience, advanced data modeling and warehouse expertise, and strong PySpark, SQL, and Python skills.
Airbnb
United States
Staff Software Engineer leading development of data catalog and metadata infrastructure for discovery, governance, lineage, and quality across Airbnb’s data ecosystem. Requires 9+ years of software engineering experience focused on data infrastructure and strong programming and distributed data technology skills.
Anyscale
San Francisco, CA
Staff Software Engineer responsible for designing and scaling Ray Data’s distributed data-processing infrastructure for large-scale AI training and inference. Requires 6+ years of production software and architectural ownership experience, plus deep distributed-systems expertise and strong Python skills.
United States
Leads the design, operation, and technical direction of Pinterest’s data workflow and context control planes, driving reliability, scalability, AI-native capabilities, and open-source contributions. Requires 10+ years of distributed-systems experience, infrastructure expertise, and proficiency in Python or Java.
Mozilla
United StatesLeads the design and improvement of large-scale data platform systems and workflows, collaborating across engineering, data science, and business teams. Requires 10+ years of relevant experience, advanced SQL and Python, cloud data tooling, and technical leadership.