AI systems do not behave like traditional software. Their outputs can vary, retrieval can fail silently, agents can choose the wrong tool, and a passing functional check can still hide serious quality or safety risk. This course gives QA professionals a practical framework for testing those systems with confidence. You will move from familiar software-testing concepts into modern AI evaluation. The course begins by translating assertions, test data, acceptance criteria, and regression testing into an eval-driven approach. You will then test LLM outputs for correctness, relevance, consistency, safety, and non-deterministic behavior using deterministic checks, model-graded evaluation, and human review.
Using a runnable QA Shop Assistant sample application, you will investigate real failure modes instead of relying only on theory. You will build evaluation datasets, validate Retrieval-Augmented Generation (RAG), measure retrieval quality, test faithfulness and grounding, and catch hallucinations. You will also test agent behavior, including tool selection, tool arguments, returned results, multi-step trajectories, and permission boundaries.
The security section covers the expanded attack surface of AI applications, prompt injection, red-team thinking, Promptfoo-based checks, and safety and policy evaluations. You will then bring these practices into delivery pipelines by running evals in GitHub Actions, defining thresholds and quality gates, identifying regressions, and planning production monitoring.
