Data Annotation Specialist, Software Engineering
Evaluates and improves AI-generated software engineering outputs by testing coding capabilities, reviewing code and agent trajectories, and reporting quality trends. Requires software engineering expertise, 3–5 years of industry experience, proficiency in several programming languages, and API development experience.
About the job
Responsibilities
- Evaluate the model's ability to respond to coding requests, workflows, and codebase-related questions using available tools.
- Assess agent trajectories and model capabilities for code generation and debugging requests.
- Prompt models to complete complex coding tasks and review the accuracy of generated responses.
- Label, proofread, and improve machine-written and human-written software engineering outputs.
- Report quality and performance trends related to model and agent behavior and project assignments.
Requirements
- Subject matter expertise in software engineering, including formal study in the field.
- 3–5 years of relevant industry experience.
- Proficiency in Python, Java, JavaScript, Go, and SQL.
- Experience designing, building, and deploying APIs.
- Ability to commit to a minimum of 16 hours per week.
- A personal laptop for the BYOD arrangement.
Nice-to-Haves
- Experience working with code agents such as OpenCode, Claude Code, Codex, or Cursor.
- Experience evaluating agent trajectories.
Compensation and Contract Details
- Part-time independent contractor position.
- 40 CAD/hour contract rate.
- 12-month contract.
- Performance incentives included.
- Remote work within Canada.
- Contractors are not eligible for employee health benefits or other employee benefits.
- Continuous work is not guaranteed; workload and compensation may fluctuate based on project needs and availability.
- The role may involve exposure to explicit, offensive, or otherwise inappropriate text content during annotation tasks.
Skills
Python, Java, JavaScript, Go, SQL, API Design, Api Deployment, Code Generation, Code Debugging, Opencode, Claude Code, Codex, Cursor
Similar jobs
Data Analytics jobs
PointClickCare
Mississauga, Canada
Leads qualitative and quantitative market research in healthcare, translating customer, market, and competitive data into recommendations for product, marketing, and growth strategy. Requires five years of healthcare market research experience, advanced analytics, strong presentation skills, and proficiency with Excel, PowerPoint, Power BI, and AI solutions.
Virta Health
Remote
The Data Analyst supports Product Analytics and the Client Group by reconciling cross-system billing data, resolving discrepancies, and building durable data-quality controls and reporting. The role requires advanced Excel, strong SQL, and 5+ years of analytics, data, or finance-operations experience.
Dropbox
United StatesAnalyzes Talent Acquisition and workforce data to identify trends, improve hiring effectiveness, and inform People and business-leader decisions. The role requires 3+ years of HR-focused analytics experience, SQL and advanced Excel proficiency, and strong stakeholder communication.
Kraken
CanadaValidates and monitors AML/CFT transaction monitoring, customer risk assessment, screening, and AI models while assessing regulatory compliance, model risk, performance, and bias. Requires a quantitative bachelor's degree, 3+ years of relevant experience, and proficiency in Python, R, and SQL.
Stripe
Canada
The role partners with cross-functional teams to translate business needs into data problems, build pipelines, metrics, dashboards, and reports, and deliver actionable recommendations. Candidates need a relevant bachelor’s degree with three years of experience or a master’s degree with two years, plus SQL proficiency.