Principal Engineer, CAPE

Charter — The Problems You'll Own

• Unified observability plane correlating GPU, networking (InfiniBand/RoCE), storage, orchestration, and workload signals for fast diagnosis and recovery.
• Treat the fleet as one logical computer: tens of thousands of accelerators with one health model, scheduler, and source of truth.
• Closed-loop autonomy: diagnose, decide, and remediate (drain, checkpoint, replace, resume) with no human in the loop, earning trust to act on live jobs.
• Maximize goodput as the objective function, trading scheduling, placement, and maintenance decisions against it.
• Predict failures hours ahead using ML on noisy hardware telemetry (GPU, NVLink, optics, thermal) and pre-emptively migrate work.
• Straggler & silent-failure detection: isolate exact rank, GPU, and node from collective-operation signals.
• Energy-aware compute: schedule, throttle, and place workloads against real-time energy availability, cost, and thermal headroom.
• Digital twin of the fleet to simulate failures, scheduling policies, and remediation logic safely.
• Agentic operations: an agent that proposes and executes fixes with guardrails and audit trails.
• Zero-trust, fully auditable multi-tenancy with identity-scoped, policy-checked actions.
• Self-qualifying hardware: automated burn-in for new/repaired nodes before taking load.

What You'll Bring

• 10+ years building infrastructure-layer systems at scale (fleet management, distributed control planes, scheduler internals, or hardware lifecycle automation).
• Deep experience with distributed systems design: consensus, state reconciliation, closed-loop automation, and autonomous decision-making on live production infrastructure.
• Hands-on fluency with GPU/HPC infrastructure, including GPU health telemetry, NVLink/InfiniBand/RoCE fabrics, thermal and power behavior.
• Track record designing and shipping large-scale observability or telemetry platforms correlating signals across compute, network, and storage.
• Comfort operating in ambiguity and defining architecture/standards for 0→1 systems.
• Strong software engineering fundamentals in at least one systems language (Go, Rust, C++, or similar) with judgment on build vs. adopt.
• Experience applying ML/statistical methods to noisy operational telemetry (failure prediction, anomaly detection) is a strong plus.
• Prior exposure to zero-trust or policy-based multi-tenancy architectures is a plus.

Benefits

• Competitive compensation and equity packages including Restricted Stock Units.
• Paid time off, paid holidays & leave of absence programs.
• Comprehensive health, dental & vision insurance; employer contributions to HSA.
• Paid parental leave; paid life insurance, short-term and long-term disability.
• Professional development & tuition reimbursement.
• Mental health & wellness support.
• Commuter benefits (parking & transit); cell phone stipend.
• 401(k) Retirement plan with company match up to 4% of salary.
• Volunteer time off; global travel insurance & emergency assistance.
• Daily meals allowance; additional perks & programs specific to location.