Senior Software Engineer – Foundational Data Systems for AI

What You’ll Build

• Global Metadata Substrate: Architect the transactional and metadata substrate that supports time-travel, schema evolution, and atomic consistency across petabyte-scale tabular datasets.
• Adaptive Engines: Build systems that reorganize data autonomously, learning from access patterns and workloads to maintain peak efficiency without manual tuning.
• Intelligent Data Layouts: Optimize bit-level organization (encoding, compression, layout) to extract maximal signal per byte read.
• Autonomous Compute Pipelines: Develop distributed compute systems that scale predictively, adapt to dynamic load, and maintain reliability under failure.
• Research to Production: Implement new algorithms in compression, representation, and optimization emerging from ongoing research. Opportunities to publish and open-source are encouraged.
• Latency as Intelligence: Design for minimal time between question and insight, enabling models and humans to learn faster from data.

What You Bring

• Depth in distributed systems: consensus, partitioning, replication, fault tolerance.
• Experience with columnar formats such as Parquet or ORC and low-level encoding strategies.
• Understanding of metadata-driven architectures and adaptive query planning.
• Production experience with Spark, Flink, or custom distributed engines on cloud object storage.
• Proficiency in Java, Rust, Go, or C++ with an emphasis on clarity and quality.
• Curiosity about theory of the mathematics of compression, entropy, and learning efficiency.
• A builder’s mindset: pragmatic, rigorous, and grounded in long-term systems thinking.

Bonus

• Familiarity with Iceberg, Delta Lake, or Hudi.
• Research or open-source contributions in compression, indexing, or distributed computation.
• Interest in how data representation affects training dynamics and model reasoning efficiency.

Compensation & Benefits

• Competitive salary, meaningful equity, and substantial bonus for top performers.
• Flexible time off plus comprehensive health coverage for you and your family.
• Support for research, publication, and deep technical exploration.