Builds measurement, evaluation, and feedback infrastructure for AI agent quality. Designs datasets, pipelines, dashboards, and analysis tools to improve agent reliability, partnering with research and product teams. Requires strong data acumen and software engineering skills.
Salary not listed
On-siteData Engineering
About the role
Responsibilities
Design and build best-in-class AI evaluation system: curated datasets, offline replay, scorers / judges, regression alerts, and dashboards.
Design feedback loops from real usage: collecting, cleaning, and interpreting user signals to inform model and harness changes.
Develop analysis tooling and workflows for debugging agent behavior: deep dives on failure modes, clustering themes, and surfacing actionable insights.
Improve reliability and guardrails by making quality measurable and operational: defining “good/bad/degraded” sessions, alerting, and triage primitives.
Requirements
Built and operated evaluation or measurement systems, such as AI evals, experimentation, ranking/relevance, or search quality. Can turn ambiguous “quality” questions into concrete metrics, pipelines, and decisions.
Strong data acumen, and can collaborate effectively with data scientists and researchers.
Taste and strong opinions on model and agent behaviors. Stay up-to-date on emerging research and industry trends.
Strong software engineering fundamentals and enjoy shipping production systems.
Skills
Ai EvaluationDatasetsScorersDashboardsPipelinesData AnalysisMachine LearningPythonSQLExperimentation
Senior Analytics Engineer owning OnePay's dbt models, Databricks BI, data quality, and semantic layers on a fast-moving fintech team. Requires 5+ years production analytics engineering, expert SQL/dbt, Databricks experience, and daily AI coding tool usage.
130k – 170k/yr
Remote5+ YOEData Engineering
Forward Deployed Data Engineer (Integration)
HilbertSan Francisco, CA
Forward Deployed Data Engineer building hybrid data pipelines and semantic layers for Hilbert's AI Growth Engine. Implements warehouse-native or managed ClickHouse integrations, partners with AI agents for accelerated onboarding, and ensures reasoning consistency across customer environments.
Salary not listed
HybridData Engineering
Software Engineer, Data Infrastructure
The Voleon GroupNew York, NY +1
Software Engineer building scalable data infrastructure, cataloging, versioning, and lineage tools to support ML research and production workflows at an AI-driven hedge fund. Requires 3+ years experience, strong software design skills, and expertise in a modern language like Python or Java.
235k – 300k/yr
Remote3+ YOEData Engineering
Client Delivery Specialist
Hinge HealthSan Francisco, CA
Manage end-to-end file-based data integrations, ingestion, transformation, and maintenance for eligibility, marketing, and reporting. Own data integrity, resolve issues, automate workflows with AI, and partner cross-functionally with Customer Success, Engineering, and Revenue Operations teams.
80k – 120k/yr
HybridData Engineering
Software Engineer
xAIPalo Alto, CA
Build and operate realtime and batch data pipelines processing billions of events daily at xAI. Design distributed data platforms, own data correctness, create shared datasets for product and business teams, and partner on data acquisition using tools like Spark, Kafka, Flink, and SQL.