# Data Scientist, Agent

**Company:** [Lovable](https://hotfix.jobs/companies/lovable)
**Location:** Stockholm, Sweden
**Role:** Data Science
**Skills:** SQL, Python, Llm Evaluation, Observability, Braintrust, OpenTelemetry, BigQuery, Pub/Sub, Hex, A/B Testing, Applied Statistics, GCP
**Posted:** 2026-08-06

> Owns the measurement and improvement of Lovable’s AI agent by defining quality metrics, building evaluation and experimentation systems, and turning telemetry into product fixes. The role requires strong SQL, Python, statistics, experimentation, and interest in LLM evaluation and observability.

## Job Description

## Requirements
- Experience or strong interest in LLM evaluation and observability, including building evaluations, scoring outputs, tracing agent behavior, and catching regressions.
- Strong SQL and Python skills, applied statistics, and experimentation experience.
- Comfortable designing A/B tests for agent changes with noisy outcomes.
- Ability to build systems and agents that produce continuous insight rather than one-off analyses.
- Instinct for defining and measuring good agent behavior, including success, error rates, and task completion, when there is no clean answer key.
- Entrepreneurial mindset and comfort working in ambiguity with agent engineers.

## Responsibilities
- Define and own metrics for agent quality, including success, completion, error rates, and the behaviors that drive them.
- Build evaluation systems and experiment frameworks that determine whether agent changes should ship.
- Design A/B-tested rollouts that catch increases in errors before changes reach all users.
- Turn agent traces and telemetry into concrete fixes in collaboration with the agent engineering team.
- Build tooling and agents that produce evaluations continuously as the agent evolves.
- Establish standards for judging agent behavior when there is no answer key.

## Tech Stack
- **Languages:** SQL, Python
- **LLM evaluation and observability:** Braintrust, OpenTelemetry (OTEL) tracing, LLM providers
- **Warehouse and events:** BigQuery, Pub/Sub
- **Analytics and product:** Hex, Lovable Apps
- **Experimentation:** A/B testing, growth testing
- **Cloud:** Google Cloud Platform (GCP)

## Similar jobs

- [Senior Data Scientist, Company Planning & Execution](https://hotfix.jobs/jobs/b80dde27-bfbc-48bf-a1b8-774c6c0877d8) - Spotify - Stockholm, Sweden
- [Senior Data Scientist](https://hotfix.jobs/jobs/8dcc12db-32cf-4da1-af89-25ef95de4928) - Vanta - Remote
- [Staff Data Scientist - Experience](https://hotfix.jobs/jobs/af8cb197-8b2f-4041-9721-d23e3cecfa28) - Spotify - Stockholm, Sweden

**Apply:** https://hotfix.jobs/jobs/180fb67a-a6ed-4a06-bfa8-0ae39e81085b
**Canonical:** https://hotfix.jobs/jobs/180fb67a-a6ed-4a06-bfa8-0ae39e81085b