Skip to content
FirecrawlFirecrawlSan Francisco, CA

Research Engineer – Evals

Build and own evaluation systems, metrics, pipelines, and datasets to rigorously measure the quality of Firecrawl's LLM-ready web data outputs at massive scale, driving model and product improvements.

210k – 275k
Remote4+ YOEML Engineering

About the role

What You'll Do

  • Design the metrics that define what "good output" actually means across millions of sites, formats, and edge cases
  • Build the pipelines and harnesses that measure quality rigorously and at scale
  • Generate and curate the datasets that make evaluation trustworthy
  • Own the feedback loop from output quality back to model and product decisions
  • Turn "did that work?" into an answer the whole team can act on

What We're Looking For

  • Engineering depth to build real evaluation systems, not just run existing ones
  • Care deeply about what "good" means and how to measure it rigorously
  • Comfortable owning ambiguous problems where the metric itself has to be invented
  • Move fast and close the loop - ship, measure, and iterate

Requirements

  • 4+ years in ML, research engineering, or data-heavy backend, with real evaluation work
  • Must already be authorized to work in the US or our eligible remote-hire regions

Nice-to-Haves

  • Experience designing metrics, building evaluation pipelines, generating datasets for LLM/web data quality
  • Background in measuring complex AI system outputs at scale

Compensation & Benefits

Salary: $210,000–$275,000/year (U.S.-based in San Francisco, CA; adjusted for other locations based on cost of living)
Equity: Up to 0.05%
Benefits (US-based): Full medical, dental, vision (100% for employees); life & disability insurance; 401(k); generous PTO (15+ days); 12 weeks parental leave; wellness stipend; learning stipend; pet insurance.
SF-specific: HQ perks, e-bike loaner.
Other: Sabbatical after 4 years; team offsites.

Skills

Evaluation SystemsMetrics DesignData PipelinesDataset CurationMl EngineeringResearch EngineeringLlm EvaluationQuality Measurement

Similar roles

ML Engineering jobs
Sequence

AI Engineer - NYC

SequenceNew York, NY

Build production-grade AI infrastructure and agentic systems for a revenue automation platform. Own end-to-end features from ideation through reliable deployment for business-critical finance workflows.

210k – 230k
Hybrid3+ YOEML Engineering
Atomicmachines

AI CAD Engineer

AtomicmachinesEmeryville, CA +1

Develops generative and programmatic CAD pipelines for micro-device assemblies, integrating AI-driven workflows and CAD APIs to automate design exploration and optimization. Requires CAD scripting expertise, AI knowledge, and 5+ years experience (PhD) or equivalent.

210k – 260k
On-site5+ YOEML Engineering
Layer Health

Machine Learning Engineer

Layer HealthBoston, MA +1

Designs, develops, and deploys scalable ML systems using LLMs to process clinical data for healthcare applications. Requires 5+ years backend/cloud experience, Python fluency, and familiarity with ML frameworks; works onsite in Boston or NYC.

210k – 250k
On-site5+ YOEML Engineering
Redis

Applied AI Engineer - Bay Area

RedisSan Francisco, CA

Senior Applied AI Engineer focused on helping customers build AI apps with Redis (80% customer engineering, 20% development). Requires 5+ years Python experience and 2+ years in ML systems.

211k – 264k
On-site5+ YOEML Engineering
Plaid

Machine Learning Engineer - Embedded Insights

PlaidSan Francisco, CA +2

Drive ML initiatives from concept to production on the Embedded Insights team. Identify opportunities, build and deploy models using Plaid's financial datasets, and partner with product teams to deliver scalable customer-facing intelligence products.

212k – 272k
Hybrid5+ YOEML Engineering