This course teaches you how to evaluate, test, debug, and improve AI systems built with large language models, Retrieval-Augmented Generation (RAG), and AI agents. Instead of relying on a few hand-picked prompts and deciding that an output “looks good,” you will learn how to approach AI evaluation like an engineer: define measurable criteria, build evaluation datasets, detect regressions, analyze failures, and make evidence-based decisions about your system.
You will explore why evaluating generative AI is fundamentally different from traditional software testing. LLM outputs are probabilistic, multiple answers can be valid, and a response can sound convincing while still being incorrect. Throughout the course, you will learn how to turn these challenges into structured and repeatable evaluation workflows.
A major part of the course focuses on LLM-as-a-Judge. You will learn how model-based evaluators work, where they are useful, and the biases and failure modes that can make evaluation scores misleading. You will design evaluation rubrics, compare model outputs, measure qualitative characteristics, and understand when automated evaluation should be combined with deterministic checks and human judgment.
