# Manager, AI Engineering (Tester )

Hiring organization: [Mastercard](https://career.thegoodapps.co/organizations/mastercard)

Canonical page: https://career.thegoodapps.co/jobs/4f4910f1-1a1b-4134-ac8a-7c68ca6eaf74

Listed on Mastercard's own careers site. Applications go to them directly.

- Employment type: full time
- Seniority: Manager
- Location: O Fallon, MO
- Salary: 140000 – 231000 USD per year

## Summary

This is a management role overseeing AI quality and testing for Mastercard's generative AI and LLM systems, suited to engineers with deep expertise in production AI evaluation frameworks and the ability to lead testing strategy across complex AI workstreams.

_Our summary, not Mastercard's wording._

## Skills named

Amazon CloudWatch, Amazon Web Services (AWS), Databricks, Datadog, Grafana, Hugging Face, LangChain, LangGraph, Microsoft Azure, MLflow, OpenAI, Python, SQL

## Required

- Master's or Bachelor's degree in Computer Science, AI/ML, or Software Engineering
- Extensive hands-on experience leading AI/ML quality engineering or LLM testing programs in production
- Expertise testing LLM and Gen AI systems including prompt testing, output evaluation, hallucination detection, RAG assessment, and agentic workflow validation
- Deep hands-on knowledge of AI evaluation frameworks and tooling such as RAGAS, DeepEval, TruLens, LangSmith, or PromptFlow
- Strong understanding of Gen AI failure modes and proven methods to surface and document them
- Strong Python programming skills with ability to build test automation scripts and evaluation pipelines independently
- SQL proficiency
- Working knowledge of LLM ecosystems including OpenAI, Anthropic, Hugging Face, LangChain/LangGraph
- Familiarity with MLOps/LLMOps pipelines and experience integrating automated quality gates into CI/CD workflows
- Experience with cloud AI infrastructure and observability tooling for monitoring live AI system behavior in production
- Strong analytical, communication, and stakeholder management skills

Apply on Mastercard's site: https://mastercard.wd1.myworkdayjobs.com/en-US/CorporateCareers/job/OFallon-Missouri/Manager--AI-Engineering--Tester--_R-281585-1