Data Annotation Specialist, Generalist

Responsibilities

• Evaluate and rank model outputs for accuracy, helpfulness, tone, and safety, providing clear justifications.
• Stress-test models to identify failure modes, unsafe behavior, and capability gaps, and document reproducible cases.
• Create prompts, responses, and exemplars for training and evaluation datasets.
• Build and apply grading criteria, rubrics, and taxonomies.
• Annotate, audit, and correct text, image, and structured data such as JSON, CSV/TSV, and Markdown.
• Participate in calibration exercises and inter-annotator agreement checks.
• Adapt to experimental and evolving task types.
• Report model performance trends and provide evidence-based feedback to cross-functional partners.

Requirements

• 1+ years of experience in AI data annotation, LLM evaluation, content moderation, research, or a related analytical role, with exposure to quality assurance and/or preference ranking.
• Experience applying detailed guidelines to complex and ambiguous content.
• Strong contextual and sociocultural judgment, with sensitivity to nuance, tone, and register.
• Comfort with ambiguity and willingness to flag unclear edge cases.
• Strong attention to detail, accuracy, and consistency across high-volume tasks.
• Excellent written English and reading comprehension, including the ability to justify evaluations and write clear prompts and exemplars.
• Ability to work independently in a remote, global, asynchronous environment.
• This is a part-time independent contractor position within Canada, requiring a commitment of 16 hours per week.
• BYOD; applicants must provide their own laptop.
• The work may involve exposure to explicit, offensive, or otherwise inappropriate content.

Nice-to-haves

• Familiarity with large language model behavior, including hallucination, sycophancy, and instruction-following gaps.
• Fluency in another language.
• Experience with annotation platforms and structured formats such as JSON, CSV/TSV, Markdown, XML, and YAML.

Compensation

• CAD $30/hour contract rate.
• Independent contractors are not eligible for employee health benefits or other employee benefits.
• Work is project-based, and continuous work is not guaranteed; hours and compensation may fluctuate based on project availability.