Senior Software Engineer

What You'll Do

• Define and build evaluation methodologies for Deepgram's models, spanning speech-to-text, text-to-speech, and emerging LLM, RAG, agent, and multimodal systems.
• Design, build, and maintain automated evaluation pipelines across batch and streaming (e.g. WER, runaway/hallucination detection, latency and time-to-first-byte), with a focus on correctness, reproducibility, and ease of adoption.
• Build scalable, reproducible evaluation infrastructure — harnesses, orchestration, and result-aggregation pipelines — running against production models and, where needed, large GPU clusters.
• Translate Research benchmarks and expected model metrics into automated, enforceable pass/fail gates.
• Build and operate canaries and continuous-monitoring systems that detect quality regressions in production before they reach customers.
• Partner with DevOps/Infra to stand up ephemeral test environments and results-aggregation infrastructure.
• Work alongside Research, model training, inference, and product teams to provide trusted evaluation signals that inform release and optimization decisions.
• Integrate evaluation and quality gates into CI/CD so quality is verified continuously, not manually.
• Help raise the bar through code reviews, technical design discussions, and strong engineering and QA practices.

What We're Looking For

• BS, MS, or PhD in Computer Science, AI, Applied Math, or a related field, or equivalent experience.
• 5+ years of professional software or QA engineering experience, with a track record of shipping test infrastructure or evaluation systems (senior candidates with significantly deeper experience welcome).
• Solid backend/scripting experience in a language such as Python, Rust, Go, or similar.
• Experience designing and building automated test pipelines, evaluation frameworks, or data-processing systems.
• Strong analytical skills and comfort reasoning about metrics, thresholds, and statistical variation in results — able to distinguish real regressions from noise.
• Ability to take charge of ambiguous technical challenges and communicate effectively across research, engineering, and product teams.

Nice to Have / Ways to Stand Out

• Hands-on experience evaluating modern AI systems such as LLMs, RAG pipelines, agents, or multimodal models, including model behavior analysis.
• Experience with React Native or other cross-platform mobile frameworks for building tooling that's accessible beyond the desktop.
• Experience building or improving evaluation frameworks, benchmarks, or ML infrastructure used by other teams or external users.
• A strong appreciation for evaluation quality — correctness, reproducibility, and consistency across environments.
• Experience with voice, audio, speech recognition, or real-time systems, and familiarity with metrics like WER, MOS, or latency/TTFB.
• Prior involvement in open-source projects, through contributions, reviews, maintenance, or community engagement.
• Experience acting as a technical bridge across teams or platforms (evaluation, training, inference, agent frameworks), combining architectural understanding with clear communication and influence.
• Familiarity with cloud infrastructure, containerized/ephemeral environments, and monitoring tooling (e.g. Grafana, canaries, anomaly detection).