Member of Technical Staff
Analyzes and improves search and answer quality by designing metrics, building evaluation and labeling pipelines, and validating changes through experiments. Requires 4+ years of analyst experience, production-grade coding skills, and proficiency in SQL and Python.
About the job
Responsibilities
- Find and diagnose quality issues in the search pipeline.
- Design metrics from scratch to track and measure search quality.
- Build datasets for model training, including LLM-as-a-judge labeling pipelines.
- Improve search snippet quality and page-selection algorithms for indexing.
- Design and analyze A/B experiments to validate improvements.
Requirements
- 4+ years of experience as a data analyst or in a related role.
- Strong coding skills, including the ability to write production-grade code at a mid-level backend-engineer level.
- Proficiency with SQL and Python.
- Hands-on experience with at least one of the following:
- Designing metrics from scratch.
- Building labeling pipelines using LLM-as-a-judge.
- Training machine-learning models shipped to production with measurable metric improvements.
- Designing evaluations with known ground truth, such as SimpleQA or BrowseComp, or driving meaningful improvements on such evaluations.
Preferred Qualifications
- Experience working on search-related products.
- Knowledge of statistics and A/B experiment design.
- Experience with Apache Spark or Databricks.
Skills
SQL, Python, Search Quality, Data Pipelines, Llm-As-A-Judge, Machine Learning, A/B Testing, Statistics, Spark, Databricks
Similar jobs
Data Analytics jobs
Kraken
CanadaValidates and monitors AML/CFT transaction monitoring, customer risk assessment, screening, and AI models while assessing regulatory compliance, model risk, performance, and bias. Requires a quantitative bachelor's degree, 3+ years of relevant experience, and proficiency in Python, R, and SQL.
Addepar
Edinburgh, United Kingdom
Manages alternative-investment data operations, including security-master oversight, workflow controls, document interpretation, and high-volume data validation. The role requires investment-industry knowledge, strong SQL and Databricks/PySpark skills, and the ability to troubleshoot ingestion pipelines and apply AI tooling.
Virta Health
Remote
The Data Analyst supports Product Analytics and the Client Group by reconciling cross-system billing data, resolving discrepancies, and building durable data-quality controls and reporting. The role requires advanced Excel, strong SQL, and 5+ years of analytics, data, or finance-operations experience.
Supernova Technology
London, United Kingdom
Analyzes, validates, and reports on operational data while supporting UK and European clients with data-related issues. The role requires strong SQL, BI, database, reconciliation, and communication skills, with at least two years of relevant experience.
Collectly
Belgrade, Serbia
Own product performance dashboards and metrics across healthcare revenue cycle products, using SQL, Python, and AI-assisted development to identify product issues and explain customer-facing trends. Requires 4+ years of analytics experience, strong communication, and a customer-focused product mindset.