M
$140,000 – $231,000
Listed on Mastercard’s own careers site. You apply with them directly — we never stand between you and the employer.
What this role is
This is a management role overseeing AI quality and testing for Mastercard's generative AI and LLM systems, suited to engineers with deep expertise in production AI evaluation frameworks and the ability to lead testing strategy across complex AI workstreams.
Our summary, not Mastercard’s wording. The full posting is on their site.
Skills this role names
- Amazon CloudWatch
- Amazon Web Services (AWS)
- Databricks
- Datadog
- Grafana
- Hugging Face
- LangChain
- LangGraph
- Microsoft Azure
- MLflow
- OpenAI
- Python
- SQL
Log in to see which of these are already on your profile.
What they ask for
Required
- Master's or Bachelor's degree in Computer Science, AI/ML, or Software Engineering
- Extensive hands-on experience leading AI/ML quality engineering or LLM testing programs in production
- Expertise testing LLM and Gen AI systems including prompt testing, output evaluation, hallucination detection, RAG assessment, and agentic workflow validation
- Deep hands-on knowledge of AI evaluation frameworks and tooling such as RAGAS, DeepEval, TruLens, LangSmith, or PromptFlow
- Strong understanding of Gen AI failure modes and proven methods to surface and document them
- Strong Python programming skills with ability to build test automation scripts and evaluation pipelines independently
- SQL proficiency
- Working knowledge of LLM ecosystems including OpenAI, Anthropic, Hugging Face, LangChain/LangGraph
- Familiarity with MLOps/LLMOps pipelines and experience integrating automated quality gates into CI/CD workflows
- Experience with cloud AI infrastructure and observability tooling for monitoring live AI system behavior in production
- Strong analytical, communication, and stakeholder management skills