AI Engineer, Intern

Responsibilities

Benchmarks and Evaluation

• Contribute to APIFlow-Bench, an open-source benchmark for API-development work.
• Design and review benchmark tasks and mock API environments.
• Extend the evaluation harness and task-generation pipeline in Python.
• Maintain a public multi-model leaderboard with statistical confidence intervals.
• Help build an action-level AI safety benchmark for simulated enterprise API environments.
• Design scenarios, threat models, and auditable evaluations covering prompt injection, data exfiltration, and permission overreach.

Model Training and Efficiency

• Fine-tune open-weight models for tool calling and agentic tasks using supervised fine-tuning, distillation, and reinforcement learning.
• Run training on managed platforms and self-managed cloud GPUs.
• Design rigorous experiments, evaluate training runs on benchmarks, conduct ablation studies and error analysis, track experiments, and report results including cost.
• Evaluate ultra-low-bit quantized models for on-device use and analyze differences between quantized and full-precision models.

Agent Systems and Engineering

• Help build Postman’s in-product AI agent using tool loops, multi-step execution, and checkpointing, primarily in TypeScript.
• Read open-source agent harnesses and turn findings into design specifications and prototypes.
• Document experiments, design decisions, and runbooks.
• Flag safety, fairness, and privacy concerns in model or agent behavior.

Requirements

• Currently pursuing a BS, MS, or PhD in Computer Science, Data Science, or a related quantitative field.
• Hands-on experience training or evaluating machine-learning models through coursework, research, hackathons, or internships.
• Solid Python fundamentals, including data structures, functions, and basic testing.
• Working knowledge of at least one deep-learning framework, preferably PyTorch.
• Clear written and verbal communication and strong documentation habits.

Nice-to-Haves

• Experience fine-tuning open-weight large language models using supervised fine-tuning, LoRA, reinforcement learning, or distillation.
• Experience building LLM agents, tool-calling systems, evaluation harnesses, or benchmarks.
• Experience shipping software end-to-end, including APIs, services, CLIs, Docker, CI/CD, and cloud systems.
• Interest or experience in AI safety and robustness, including red-teaming, prompt injection, agent security, fairness, or interpretability.
• Exposure to quantization, low-bit inference, or serving optimization.
• Publications, technical blog posts, ablation studies, or self-directed projects with quantified results.
• Fluency with AI coding tools such as Claude Code, Cursor, or Codex.

Compensation and Benefits

• Pay-on-performance philosophy and flexible schedule.
• Full medical coverage, flexible PTO, wellness reimbursement, and monthly lunch stipend.
• Wellness programs, team-building events, and donation matching.
• This role is based in the San Francisco Bay Area and requires working in the office five days per week.