Agent Evals Specialist (Knowledge Graph Review)
5 – 15United StatesQA EngineeringRemote
Summary
Reviews AI agent outputs processing technical documents into knowledge graphs, verifies accuracy against source material, scores performance on rubrics, and provides detailed feedback for agent improvement. Requires strong English, focus on dense content, and consistent evaluation.
About the role
Responsibilities
- Read source material and agent's output side by side to verify accurate content capture.
- Review agent's actions: what it created, changed, or omitted.
- Score rubric on accuracy, coverage, organization, and rule adherence.
- Write detailed feedback on mistakes to improve the agent.
- Submit tasks and proceed to next.
Requirements
- Strong written English.
- Ability to read dense technical content for hours without losing focus.
- Consistent scoring over time.
- Clear, specific feedback writing.
Preferred
- Prior work as AI trainer, tutor, or evaluator (e.g., Outlier, DataAnnotation, xAI).
- Background in technical writing, editing, QA, translation, paralegal, or research assistant.
- Markdown familiarity.
Skills
Knowledge GraphsAI AgentsTechnical WritingMarkdownQuality AssuranceRubric ScoringDocument ReviewAI EvaluationFeedback WritingTechnical Editing