Member of Technical Staff, Data Analysis and Evaluation
This role designs data collection and evaluation processes, analyzes dataset quality and model robustness, and trains large language models on distributed infrastructure. It requires strong software engineering, statistics, experimental design, machine learning, and cross-functional communication skills.
About the job
Responsibilities
- Design and oversee data collection tasks, including supporting human annotators and ensuring data quality.
- Develop and apply statistical methods to evaluate dataset quality and reliability.
- Analyze and assess the generalizability and robustness of machine learning systems across diverse use cases.
- Collaborate with cross-functional teams to improve dataset quality and model performance.
- Train and fine-tune large language models (LLMs) on distributed training infrastructure.
- Conduct experiments to evaluate model performance and identify areas for improvement.
Requirements
- Extremely strong software engineering skills.
- Strong expertise designing and conducting data collection tasks, including working with human annotators.
- Strong statistical skills and experience evaluating scientific experiments related to data collection and model performance.
- Experience analyzing datasets for quality, bias, and suitability for training machine learning models.
- Hands-on experience training LLMs on distributed training infrastructure.
- Familiarity with evaluating and improving the generalizability and robustness of machine learning systems.
- Proficiency in Python and machine learning frameworks such as PyTorch, TensorFlow, or JAX.
- Excellent communication skills for collaborating with cross-functional teams and presenting findings.
- One or more papers at top-tier venues such as NeurIPS, ICML, ICLR, AIStats, MLSys, JMLR, AAAI, Nature, COLING, ACL, or EMNLP.
Compensation and Benefits
- Weekly lunch stipend of $75/£75 or equivalent in local currency.
- Full health and dental benefits, including a separate mental health budget.
- RRSP matching, 401(k), or pension scheme.
- 100% parental leave top-up for up to six months for either parent.
- Annual enrichment benefits covering arts and culture, fitness and wellness, quality time, and workspace improvements.
- Education and learning stipend for conferences, courses, and coaching.
- Six weeks of paid vacation (30 working days).
- Travel budget for remote employees to visit other offices and an annual company offsite.
- Coworking benefit for employees who are not near an office.
- $500 home office stipend.
Skills
Python, PyTorch, TensorFlow, JAX, LLMs, Machine Learning, Statistical Analysis, Experimental Design, Data Collection, Distributed Training, Dataset Evaluation, Human Annotation
Similar jobs
Data Science jobs
Zoox
Boston, MA
Develop datasets, performance metrics, and statistical methods to verify and validate an autonomous driving perception stack. The role requires advanced education in a quantitative field and proficiency in Python, SQL or Spark/Scala, exploratory analysis, visualization, and statistical modeling.
Dataiku
London, United Kingdom
Build and deploy production data science solutions with customers while providing platform training, consulting, and strategic guidance. Requires 2+ years with Python, SQL, and machine learning, plus strong communication skills and familiarity with data systems.
Axle
Bethesda, MD
Conducts independent neuroscience experiments involving multi-electrode and electrophysiological recordings, analyzes data, prepares publication-ready materials, and mentors junior staff. Requires a related master's degree, three years of experience, Biology expertise, and strong data presentation skills.
Imprint
United States
Owns data science and decisioning for onboarding fraud, identity, and KYC, developing predictive models, experiments, vendor evaluations, and monitoring systems. Requires 5+ years in data science or risk analytics, strong Python and SQL, and experience with adversarial classification problems.
PrizePicks
United States
Leads the data science function responsible for sports and esports market origination, projection accuracy, and coverage strategy. Requires 5+ years in sports betting, fantasy, or gaming, strong modeling skills in SQL and Python or R, and experience scaling high-performing teams.