Machine Learning Eval Engineer

What You’ll Do

• Design, build, and maintain evaluation benchmarks that reveal where our models perform well and where they fail.
• Develop metrics, heuristics, and workflows to automatically identify new failure modes across large and messy real-world datasets.
• Partner closely with other ML engineers to turn evaluation insights into model improvements and better training priorities.
• Work hands-on with unstructured enterprise data, including PDFs, spreadsheets, and other difficult document formats, to uncover edge cases and hard examples.
• Build lightweight internal and user-facing tools, including simple interfaces in Python frameworks like Flask, to help teams inspect results, analyze model behavior, and communicate evaluation outcomes.
• Collaborate with customers and internal teams to understand real-world data needs and create bespoke benchmarks that highlight Reducto’s strengths.

You’ll Thrive Here If You

• Hold yourself to a high bar for quality and precision.
• Enjoy solving complex problems and building from first principles.
• Have strong Python skills and can independently build clean, reliable technical solutions. Bonus points for product and frontend experience!
• Are comfortable working with data infrastructure such as AWS S3 and OLAP or analytics systems like Tinybird.
• Love getting your hands dirty with unstructured data and chasing down difficult failure cases.
• Operate well in fast-changing, high-growth environments.
• Collaborate effectively across technical and non-technical teams.
• Take full ownership from strategy through execution.

Bonus points if you

• Have experience at an early-stage or high-growth startup.
• Have some background in product thinking and can build simple, polished user-facing interfaces.
• Are comfortable working directly with customers to understand their workflows and data needs.
• Have experience in AI/ML, data infrastructure, enterprise software, or document understanding systems.
• Care deeply about combining technical excellence with business impact.