AI Engineer, Internal Systems
Build verification, evaluation, and testing systems that allow AI agents to determine when work is complete and operate reliably in production. The role requires experience with internal tools, agentic coding workflows, evaluation infrastructure, and hands-on application and infrastructure development.
About the job
Responsibilities
- Build end-to-end verification loops that enable agents to determine when their work is complete.
- Create fast, reliable testing environments and convert failures and human feedback into durable evaluation signals.
- Improve how context, instructions, skills, and memory are delivered to agents, and measure what improves their performance.
- Operate the agent fleet as a production system while measuring quality, adoption, and impact.
Requirements
- Experience building internal tools adopted by other engineers and explaining how their impact was measured.
- Deep use of agentic coding tools, with informed perspectives on their strengths and limitations.
- Experience building evaluation or verification infrastructure, such as test harnesses, CI systems, evaluation pipelines, or benchmarks.
- Ability to work hands-on across application code, infrastructure, and unfamiliar systems.
- Empirical approach to work, attention to subtle failure modes, and comfort owning ambiguous problems.
Nice-to-haves
- Prior experience in voice, model training, or related product domains.
Skills
Agentic Coding, Evaluation Infrastructure, Verification Infrastructure, Test Harnesses, Ci Systems, Evaluation Pipelines, Benchmarks, Application Development, Infrastructure, Production Systems
Similar jobs
ML Engineering jobs
Cinder
New York, NY
Build and operate production machine-learning systems for content safety, from messy customer data through classification, evaluation, and inference. The role requires 5+ years of ML engineering experience, strong Python and MLOps skills, and sound judgment across classical models and LLMs.
Perplexity
San Francisco, CA
Build AI agent harnesses, models, and product capabilities that enable agents to perform complex work across digital environments. The role combines applied AI research and software engineering, requiring Python proficiency, strong product judgment, and experience with agent tooling, reinforcement learning, or browser technologies.
Firecrawl
San Francisco, CA
Build and operate production machine-learning systems for search ranking, relevance, extraction quality, and LLM-driven features. The role requires production ML ownership, ranking or relevance expertise, large-scale data experience, Python, and rigorous experimentation skills.
ClickUp
United States
Build and operate large-scale ranking and retrieval systems that power search relevance, including hybrid lexical/vector search, embeddings, query understanding, and permission-aware retrieval. Requires a bachelor's degree and 5+ years of ML engineering experience in ranking or information retrieval.
Atomicmachines
Emeryville, CA
Build and operate production ML infrastructure spanning training, deployment, serving, monitoring, data pipelines, and feedback-driven retraining. The role requires strong MLOps and DevOps experience, Python and SQL proficiency, and ownership of reliable cloud-based systems.