Staff Data Engineer, Core Migrations

What You’ll Do

• Lead discovery & technical due diligence: analyze poorly documented legacy systems (ETL, stored procedures, reporting layers, file feeds), reconstruct business logic, and capture it in lineage maps, mapping specs, and risk analyses.
• Reverse-engineer complex legacy systems using agents: reconstruct intent from undocumented systems (SSIS packages, stored procedures) to build modern equivalents.
• Drive ambiguity to resolution: identify unknowns early, pull answers from clients, SMEs, and Operations.
• Architect & build migration pipelines: convert legacy logic into production-grade Airflow DAGs and Spark jobs, handling edge cases, payer-specific rules, from ingestion through reconciliation to handoff.
• Make and document architectural decisions: own pipeline design, partitioning strategy, validation approach, and document reasoning.
• Prove correctness at scale: build automated reconciliation frameworks to ensure migrated output matches source data.
• Own the program end-to-end: scope, sequence, track work; surface risks; align cross-functional teams and client; drive UAT to sign-off.
• Raise the bar for the practice: codify runbooks and retrospectives, mentor engineers, turn pain points into reusable tooling.

What You Bring

• 8+ years as a hands-on Data Engineer or Software Engineer, independently owning complex, multi-stakeholder technical projects.
• Strong Python and SQL, including complex legacy code.
• Deep Apache Spark expertise: distributed processing, performance tuning, partitioning, debugging at scale.
• Advanced Apache Airflow: designing and authoring production DAG architectures from scratch.
• AI-assisted development: actively uses coding agents (Claude, Copilot, Cursor); prompt-engineering for code, debugging, documentation; critically evaluates AI output.
• Experience with legacy ETL stacks: reconstruct intent from SSIS, SQL Server stored procedures, T-SQL and translate to modern stack.
• Data validation & reconciliation at scale: designing automated frameworks for high-confidence data matching.
• Proficient with AWS: S3, partitioned object storage, secure cross-account access, cost/performance trade-offs.
• Cross-functional coordination across engineering, operations, and client-facing teams.
• Clear technical communication: writing discovery specs, architecture docs, stakeholder updates for technical and non-technical audiences.
• Data-first mindset with experience building automated reconciliation frameworks guaranteeing data fidelity.