# Senior Software Engineer

**Company:** [Deepgram](https://hotfix.jobs/companies/deepgram)
**Location:** Remote
**Role:** ML Engineering
**Salary:** $180k – $240k/yr
**Experience:** 5+ years
**Skills:** Python, Rust, Go, LLMs, RAG, wer, mos, Grafana, React Native, CI/CD
**Posted:** 2026-07-28

> Senior Software Engineer building automated evaluation pipelines, test infrastructure, and monitoring systems to validate quality of Deepgram's speech, audio, LLM, and multimodal AI models before production release. Requires 5+ years building test/evaluation frameworks, strong analytical skills, and backend experience in Python/Rust/Go.

## Job Description

## What You'll Do
- Define and build evaluation methodologies for Deepgram's models, spanning speech-to-text, text-to-speech, and emerging LLM, RAG, agent, and multimodal systems.
- Design, build, and maintain automated evaluation pipelines across batch and streaming (e.g. WER, runaway/hallucination detection, latency and time-to-first-byte), with a focus on correctness, reproducibility, and ease of adoption.
- Build scalable, reproducible evaluation infrastructure — harnesses, orchestration, and result-aggregation pipelines — running against production models and, where needed, large GPU clusters.
- Translate Research benchmarks and expected model metrics into automated, enforceable pass/fail gates.
- Build and operate canaries and continuous-monitoring systems that detect quality regressions in production before they reach customers.
- Partner with DevOps/Infra to stand up ephemeral test environments and results-aggregation infrastructure.
- Work alongside Research, model training, inference, and product teams to provide trusted evaluation signals that inform release and optimization decisions.
- Integrate evaluation and quality gates into CI/CD so quality is verified continuously, not manually.
- Help raise the bar through code reviews, technical design discussions, and strong engineering and QA practices.

## What We're Looking For
- BS, MS, or PhD in Computer Science, AI, Applied Math, or a related field, or equivalent experience.
- 5+ years of professional software or QA engineering experience, with a track record of shipping test infrastructure or evaluation systems (senior candidates with significantly deeper experience welcome).
- Solid backend/scripting experience in a language such as Python, Rust, Go, or similar.
- Experience designing and building automated test pipelines, evaluation frameworks, or data-processing systems.
- Strong analytical skills and comfort reasoning about metrics, thresholds, and statistical variation in results — able to distinguish real regressions from noise.
- Ability to take charge of ambiguous technical challenges and communicate effectively across research, engineering, and product teams.

## Nice to Have / Ways to Stand Out
- Hands-on experience evaluating modern AI systems such as LLMs, RAG pipelines, agents, or multimodal models, including model behavior analysis.
- Experience with React Native or other cross-platform mobile frameworks for building tooling that's accessible beyond the desktop.
- Experience building or improving evaluation frameworks, benchmarks, or ML infrastructure used by other teams or external users.
- A strong appreciation for evaluation quality — correctness, reproducibility, and consistency across environments.
- Experience with voice, audio, speech recognition, or real-time systems, and familiarity with metrics like WER, MOS, or latency/TTFB.
- Prior involvement in open-source projects, through contributions, reviews, maintenance, or community engagement.
- Experience acting as a technical bridge across teams or platforms (evaluation, training, inference, agent frameworks), combining architectural understanding with clear communication and influence.
- Familiarity with cloud infrastructure, containerized/ephemeral environments, and monitoring tooling (e.g. Grafana, canaries, anomaly detection).

## Similar roles

- [ML Infrastructure Engineer](https://hotfix.jobs/jobs/60a4dff1-7af4-474a-8a5b-b9251cf4f4b8) - xAI - Palo Alto, CA - $180k – $440k/yr
- [Software Engineer, ML Data](https://hotfix.jobs/jobs/c04a70cf-91a2-4b7d-9504-b41b76234939) - Liftoff - San Francisco, CA - $180k – $230k/yr
- [Senior Software Engineer, AI](https://hotfix.jobs/jobs/b5ff4a30-137e-4cb9-93aa-59afeae0fa12) - Ironclad - San Francisco, CA - $180k – $220k/yr
- [Senior Machine Learning Engineer](https://hotfix.jobs/jobs/4713b492-d636-407a-bb9b-2a7357fb074a) - Teleskope - New York, NY - $180k – $210k/yr
- [Senior Machine Learning Engineer](https://hotfix.jobs/jobs/cc221086-96c9-44b8-9848-3fc8a12e202b) - Arcade - San Francisco, CA - $180k – $300k/yr

**Apply:** https://hotfix.jobs/jobs/f3c2480b-2287-4246-9415-4d3a727e5994
**Canonical:** https://hotfix.jobs/jobs/f3c2480b-2287-4246-9415-4d3a727e5994