R
$240,000 – $290,000
Listed on Runway’s own careers site. You apply with them directly — we never stand between you and the employer.
What this role is
This role leads the design and development of an end-to-end model evaluation platform that will measure the quality of frontier AI models across the organization. You'll work embedded with research teams to build the infrastructure, metrics, and tooling that determines which checkpoints to train, which models to ship, and how to scientifically validate improvements.
Our summary, not Runway’s wording. The full posting is on their site.
Skills this role names
Log in to see which of these are already on your profile.
What they ask for
Required
- 5+ years building ML infrastructure or data platforms in production, including work on evaluation, experimentation, or benchmarking systems
- Strong Python and PyTorch with hands-on experience running large batch GPU workloads on Kubernetes
- Experience designing data pipelines and storage for large volumes of media or model outputs with attention to versioning and reproducibility
- Understanding of experimental statistics: paired comparisons, confidence intervals, multiple-comparison pitfalls, inter-rater agreement
- Ability to build internal tools end to end from CLI to browser
- Can lead a broad technical area by gathering requirements, setting direction, making tradeoffs, and driving roadmaps
- Familiarity with the full model development lifecycle: data, training, evaluation, serving
- Self-starter who works embedded with research teams and moves fast
- Strong systems thinking and pragmatic approach to production reliability
Nice to have
- Track record of building evaluation suites for generative models
- Hands-on work with LLM- or VLM-as-judge pipelines
- Experience with online experimentation platforms and connecting offline metrics to product outcomes
- Prior work evaluating agents or robotics policies