# Senior Software Engineer, AI Evals

Hiring organization: [Sentry](https://career.thegoodapps.co/organizations/sentry)

Canonical page: https://career.thegoodapps.co/jobs/8a96a243-dc17-4ec8-8848-0d01f16c4e56

Listed on Sentry's own careers site. Applications go to them directly.

- Seniority: Senior
- Location: San Francisco, CA
- Remote: yes
- Salary: 155000 – 400000 USD per year

## Summary

This role involves building evaluation infrastructure for Sentry's AI systems, designing datasets, benchmarks, and test harnesses that measure the accuracy and reliability of AI-powered debugging agents. It suits experienced software engineers who enjoy translating ambiguous AI behavior into concrete metrics and want to build foundational tools that enable faster, more confident AI development.

_Our summary, not Sentry's wording._

## Skills named

Agentic systems, Machine Learning, Python, TypeScript

## Required

- 5+ years of professional software engineering experience
- Bachelor's degree in computer science, machine learning, or related field
- Production-quality code in Python and TypeScript
- Experience building testing, evaluation, or data infrastructure for complex systems
- Experience with structured and unstructured datasets, labeling workflows, or data quality pipelines
- Familiarity with modern ML systems and evaluation techniques

## Nice to have

- AI/ML infrastructure experience
- Experience evaluating LLMs or agentic systems
- Experience with AI-assisted developer tools

Apply on Sentry's site: https://jobs.ashbyhq.com/sentry/95d2eeab-291d-40ad-97a2-86b104f3c7ad/application