Data Scientist, Agent
Owns the measurement and improvement of Lovable’s AI agent by defining quality metrics, building evaluation and experimentation systems, and turning telemetry into product fixes. The role requires strong SQL, Python, statistics, experimentation, and interest in LLM evaluation and observability.
About the job
Requirements
- Experience or strong interest in LLM evaluation and observability, including building evaluations, scoring outputs, tracing agent behavior, and catching regressions.
- Strong SQL and Python skills, applied statistics, and experimentation experience.
- Comfortable designing A/B tests for agent changes with noisy outcomes.
- Ability to build systems and agents that produce continuous insight rather than one-off analyses.
- Instinct for defining and measuring good agent behavior, including success, error rates, and task completion, when there is no clean answer key.
- Entrepreneurial mindset and comfort working in ambiguity with agent engineers.
Responsibilities
- Define and own metrics for agent quality, including success, completion, error rates, and the behaviors that drive them.
- Build evaluation systems and experiment frameworks that determine whether agent changes should ship.
- Design A/B-tested rollouts that catch increases in errors before changes reach all users.
- Turn agent traces and telemetry into concrete fixes in collaboration with the agent engineering team.
- Build tooling and agents that produce evaluations continuously as the agent evolves.
- Establish standards for judging agent behavior when there is no answer key.
Tech Stack
- Languages: SQL, Python
- LLM evaluation and observability: Braintrust, OpenTelemetry (OTEL) tracing, LLM providers
- Warehouse and events: BigQuery, Pub/Sub
- Analytics and product: Hex, Lovable Apps
- Experimentation: A/B testing, growth testing
- Cloud: Google Cloud Platform (GCP)
Skills
SQL, Python, Llm Evaluation, Observability, Braintrust, OpenTelemetry, BigQuery, Pub/Sub, Hex, A/B Testing, Applied Statistics, GCP
Similar jobs
Data Science jobs
Spotify
Stockholm, SwedenSenior Data Scientist helping shape company planning and execution through statistical analysis, machine learning, data modeling, and decision-focused recommendations. Requires 5+ years of quantitative experience, strong SQL and Python skills, and the ability to communicate insights to varied stakeholders.
Vanta
Remote
The Senior Data Scientist will build scalable product analytics assets, measurement frameworks, and AI-assisted analytical workflows while partnering with cross-functional teams to guide product and business decisions. The role requires 4+ years of applied data experience and strong SQL, programming, statistics, machine learning, and experimentation skills.
Spotify
Stockholm, SwedenStaff Data Scientist leading product and content-data analysis, evaluation frameworks, metrics, and self-serve analytical foundations for content safety and taxonomy. The role requires deep product data science experience, strong statistics, SQL and Python proficiency, and cross-functional leadership.