Machine Learning Researcher - RL and Agentic Systems

What You’ll Do

Design and build datasets, tasks, and environments

• Design and build datasets, tasks, environments, and evaluation assets for benchmarking agentic systems and multi-step model behavior.
• Translate real-world workflows into structured tasks, interaction traces, trajectories, stateful environments, and verifiable outcomes that can be used to evaluate advanced AI systems.

Develop frameworks for evaluating real-world data quality

• Develop frameworks that assess diversity, realism, coverage, fidelity, informativeness, and downstream usefulness of datasets for agentic systems.
• Build quality scorecards and evaluation methods that make dataset strengths, weaknesses, and failure modes legible across teams.

Benchmark model behavior in RL and agentic settings

• Evaluate planning, tool use, robustness, recovery from failure, task completion, and generalization behavior in RL-style or agentic environments.
• Connect model failures back to concrete dataset, environment, or task-design gaps and recommend improvements grounded in empirical evidence.

Build scalable evaluation and validation tooling

• Contribute to tools and systems that automate dataset validation, environment generation, rollout analysis, benchmark construction, and evaluation workflows.
• Improve internal infrastructure for reproducible experimentation, benchmark management, and evaluation quality.

Partner across research, engineering, and product

• Collaborate closely with research and engineering teams to identify data bottlenecks, improve evaluation methodology, and shape internal best practices around task-grounded AI training data.
• Represent DataLab’s perspective in cross-functional discussions around dataset quality, benchmark design, and frontier agentic-system evaluation.

What Success Looks Like

Near-term: establish a strong evaluation baseline

• Create clear benchmark frameworks, evaluation assets, and dataset-quality scorecards that help Protege reason about how real-world data impacts advanced agentic systems.
• Use rigorous evaluation methods to identify meaningful dataset improvements, improve benchmark fidelity, and sharpen the company’s understanding of what high-impact agentic data actually looks like in practice.

What You Bring

• PhD or equivalent Master’s Degree + 4+ years industry experience in machine learning, computer science, statistics, engineering, mathematics, economics, or related quantitative fields.
• Strong understanding of AI model training pipelines, evaluation methodology, and the role of data in shaping model performance.
• Experience working with large, unstructured, or semi-structured datasets used to train or evaluate ML systems.
• Experience with reinforcement learning, sequential decision-making, agentic systems, tool-using models, or multi-step model evaluation.
• Experience designing tasks, benchmarks, environments, simulations, or evaluation frameworks for real-world model behavior.
• Strong intuition for realism, coverage, difficulty, fidelity, and meaningful outcome structure in datasets.
• Strong experimental design, evaluation, benchmarking, and data-validation skills.
• High ownership and ability to independently identify and solve high-impact problems.

Nice to have

• Experience developing evaluation frameworks or performance metrics for datasets, agentic systems, or training data.
• Experience translating real-world workflows into structured tasks or environments for model evaluation.
• Experience with RLHF, RLAIF, imitation learning, reward modeling, online or offline RL, or related methods.
• Experience with Harbor or other agent evaluation frameworks.
• Publications or open-source contributions in reinforcement learning, agents, evaluation, or data-centric AI.
• Experience collaborating cross-functionally with product, infrastructure, or partnership teams.
• Experience with synthetic data generation, trajectory generation, or simulation-based environments.